SAR colorization aims to enrich gray-scale SAR images with color while ensuring the preservation of original radiometric and spatial details. However, researchers often limit themselves to using only the red, green, and blue bands of a multispectral image as the source of color information, coupled with a single-polarization channel from the SAR image. This approach neglects the intrinsic characteristics of remote sensing data and thus fails to fully leverage available information. To overcome this limitation, this research attempts to explore inclusion of all available bands from multispectral images along with dual-polarization channels from SAR imagery in the colorization process. Furthermore, we present a new colorization method called improved conditional generative adversarial network for SAR colorization (IcGAN4ColSAR). This method tries to include the spectral angle mapper index within its loss function. Sufficient experiments show that our explorations in the number of data channels and the loss function are helpful in improving the colorization performance of the SAR image.

Due to the reliability of data acquisition, synthetic aperture radar (SAR) sensors are fundamental for remote sensing applications with the need for flexibility and fast response. For urban applications, besides the analysis of salient point signatures, extracted height information allows to evaluate the state of buildings. Recently developed deep learning approaches enable height estimates in situations where only one SAR image of an area of interest is available. However, building shapes still exhibit low quality in the resulting digital surface models (DSMs). This paper presents how derived surface models from the SAR image can be refined with knowledge about the shape of buildings. For that purpose, building representations are learned with a neural network from optical images and CityGML models. The results demonstrate that our model not only effectively transfers knowledge to process DSMs from various data sources but also showcases the ability to generalize across different regions.

This study aims to advance the application of state-of-the-art Synthetic Aperture Radar (SAR)-altimetry for inland purposes, focusing on the interaction of signals with arid surfaces. A critical parameter in this exploration is the backscatter coefficient. Our analysis encompasses a thorough examination of C- and Ku-Band backscatter coefficients across diverse land cover types in Africa and South America. Emphasizing the temporal aspect, we assess the behavior of backscatter coefficients over the course of a year. The sensitivity of the backscatter coefficient to surface moisture content reveals distinct patterns associated with varying land cover types. Additionally, our findings contribute significantly to the understanding of SAR-Altimetry signals over non-water surfaces, offering valuable insights into the dynamic nature of these signals over time.

As the impact of modern human activities on ecosystems intensifies, it becomes increasingly important to accurately assess this influence. Earth Observation, particularly through satellite imagery, serves as a key environmental conservation tool, providing a comprehensive overhead perspective for monitoring our planet’s ecosystems. This study formulates a pixel-wise regression task, guided by a novel set of naturalness annotations that quantify the modern human pressure on a landscape at the pixel level. We introduce a tailored framework that integrates geographical and contextual priors. These priors are represented by coordinate information and broader contextual information surrounding the immediate patch. Our approach improves the deep neural network’s capability to estimate naturalness from satellite imagery, enabling a deeper comprehension that promotes the safeguarding of our natural habitats.

Accurate land cover maps provide crucial information for various purposes. Due to their global accessibility, and high temporal, spectral, and spatial resolution, remote sensing images are a popular source to extract and update land cover maps. However, a reliable and effective global model that inputs satellite images and outputs accurate land cover class for each pixel is complex to achieve. The reason is the varying characteristics and heterogeneous distribution of land cover classes over the globe. In this paper, we propose a geolocation-aware architecture to classify land covers globally using Sentinel-2 data. While the proposed method is trained with a global dataset, it takes the regional characteristics of the data into account by dedicating branches to each major climate zone. The Sentinel-2 images together with the corresponding ESA world cover labels from the SEN12TP dataset are used to train the network. Our proposed model is compared against a globally trained UNet, and the results reveal faster convergence and enhanced visual predictions.

In recent decades, rapid urbanization in India, fueled by population growth, has spurred the construction of new cities and the expansion of existing urban centers, extending into larger peripheral areas—a trend common in developing nations globally. Despite its widespread occurrence, accurately mapping human settlements and building distributions remains a challenge. State authorities and private enterprises, including Microsoft and Google, have sought to comprehensively capture this data. While existing initiatives offer a global perspective, challenges persist, especially in precision, for countries like India where cities boast highly dense and mixed urban development. This study explores the potential of Sentinel-2 images for building footprint mapping and change detection at 2.5 m spatial resolution, focusing on the dynamic Delhi National Capital Region (NCR). Recent works demonstrate sub-pixel accuracy in deriving building footprint maps through deep learning on Sentinel-2 imagery. The research aims to develop on existing findings taking into account seasonal variations and using improved training labels to further extend these findings to India.

This study presents a novel pretraining approach for self-supervised learning on optical Earth observation satellite data based on the masked autoencoder paradigm. Unlike typical methods limited to a single sensor’s data, our method operates across various sensors by encoding physical sensor parameters into the learning step, to account for the unique differences among sensor designs. This enables merging training datasets acquired with different sensors as well as performing inference in a sensor-independent manner. Successful encoding of the sensor parameters through our approach is shown through testing on a downstream land-cover mapping task, where baseline models are outperformed by up to 6 points for the F1-score.

Investigating the synergy of deep learning and high-resolution Synthetic Aperture Radar (SAR) data, this paper focuses on building footprint extraction – a domain traditionally dominated by optical imagery. The proposed method involves projecting SAR data onto a digital terrain model and utilizing a modified U-Net for segmenting the building outlines in a common projected map system. An extensive data set consisting of TerraSAR-X images and OpenStreetMap building footprints was created to train the model. With the very promising results, the study positions SAR as a reliable alternative for accurate building footprint mapping, with implications for time-critical disaster management and urban monitoring.

Building damage assessment is critical in regions facing geopolitical challenges. This paper explores the use of Synthetic Aperture Radar remote sensing data, specifically from the Sentinel-1 constellation, to improve the accuracy and operational efficiency of building damage assessment. The approach is based on the use of SAR backscatter time series for a pixel-wise change detection methodology. As a case study, we consider the state of Ukraine, which has experienced significant building damage due to the ongoing Russo-Ukrainian war. Using the presented approach, we demonstrate the feasibility of change detection from freely available SAR data with weekly temporal resolution and at a nationwide spatial scale.

Advancements in Earth observation (EO) have led to an increase in the volume of and easier access to multimodal geospatial data, making environmental monitoring and analysis more accessible. However, understanding the influence of each input modality on decision-making within deep learning models remains an open challenge. This letter proposes a deep occlusion framework to enhance the interpretability of a multimodal model for land naturalness assessment, using a supervised pixelwise regression task for naturalness mapping with the input modalities Sentinel-2 and Sentinel-1 imagery, land cover maps, and nighttime lights intensity data. The proposed framework systematically occludes individual input modalities to create modality-level influence scores. Influence scores are attributed to input modalities by measuring the distance between the embedding of the nonoccluded input and the embedding of the input with a single modality occluded, revealing how each modality influences predictions and clarifying their contributions (and, thus, importance) in the model’s decision-making process. The results provide further insights into how input modalities influence the model’s decision-making at both the sample level, enabling regional case studies, and the dataset level, allowing for data pruning and improving training and inference times. The code is available athttps://github.com/burakekim/embedding_occlusion.

In this research, we focus on the classification of land cover types using radar altimetry data and evaluate the sensitivity of the altimetry signal across different land cover categories. To perform the classification task, we create a comprehensive dataset by combining altimetry footprints and the ESA World-cover2020 dataset. To model the classification, we employ multiple 1D-CNN (Convolutional Neural Network) architectures originally developed for other applications and adapt them to the peculiarities of altimetry waveforms. To evaluate the performance of our approach, we employ the F1-score metric, which provides a balanced measure of precision and recall. Our experimental results demonstrate a notable F1-score of 0.86, indicating the effectiveness of our proposed method in accurately classifying land cover types from altimetry data.

In order to gain a better understanding of disturbances (i.e., anthropogenic pressure) in our environment, researchers have worked on methods for the mapping of wilderness areas given their crucial role in providing native habitat for many species, which are often endangered. In this work, we formulate the wilderness mapping task as a supervised learning problem. We focus on the joint use of potentially complementary features provided by multi-modal input data. Until now, the individual influences of different input modalities on the decision of a deep neural network have largely remained unclear. Therefore, we develop a framework for the modality-level interpretation of multi-modal Earth observation data in an end-to-end fashion. While leveraging an explainable machine learning method, namely Occlusion Sensitivity Maps, the proposed framework investigates the influence of modalities in an early-fusion setting, i.e. the modalities are fused before the learning process. With respect to the application, our results indicate that auxiliary data such as land cover and nighttime light data are important sources for the accurate classification of wilderness areas and the influence of a modality increases with the increasing number of spectral channels.

The georeferencing of images acquired from CubeSats with inaccurate navigating systems often needs to be improved by post-processing. In applications such as forest fire early warning systems, fire pixels are detected using satellite thermal images. To avoid overheads of post-processing, in this case, time complexity, it is significantly helpful to georeference the thermal images directly on the satellite. To do so, we propose a pipeline that first segments a single thermal band image into binary and multi-class land cover maps. Thereafter, it matches the predicted land cover maps to precisely geo-referenced land cover images. Our experiments indicate that the proposed deep-learning-based pipeline is able to precisely georeference 75% of the test images.

In this work, we investigate two critical design decisions that arise when adapting the concept of contrastive learning to optical earth observation data. We work within the framework of SimCLR in order to pre-train neural network architectures and test their respective applicability on downstream datasets over various common remote sensing tasks. During the training, we focus in detail on the concept of the creation of positive and negative pairs due to the introduction of different batch sampling strategies and color-related augmentations. We report all performance metrics as a function of the available training data and discuss underlying mechanisms in order to drive the understanding of contrastive learning in optical Earth observation forward.

In the field of urban monitoring, accurate mapping of building structures and the corresponding changes is one of the most essential pieces of information. In many cases, this problem is approached with very high-resolution images, needed due to the spatial complexity in urban environments. And still, many binary change detection (CD) methods cannot segregate the changes introduced by the generation of new building structures from seasonal changes or other semantic changes. In this paper, we investigate a simple approach for building CD from freely available Sentinel-2 images, that neither depends on high-resolution imagery nor is prone to be negatively affected by seasonal changes. The proposed approach is simple and mainly based on the utilization of Sentinel-2 and existing corresponding building footprint information. We discuss in detail all the necessary steps to produce sub-resolution CD maps and shine a light on the critical influence of georeferencing correction.

The deep learning-based estimation of topographic heights from single remote sensing images has shown great potential in recent years. Drawing inspiration from the computer vision task of single image depth estimation, the translation from the input remote sensing image to a height image via convolutional neural networks lies at the core of the approaches published so far. Most of the existing works, however, neglect the fact that remote sensing data comes from well-calibrated sensors carried by satellites flying in well-controlled orbits. Thus, a lot of high-quality meta-information is available for most remote sensing images, which can be used to provide the pure deep neural network with physically meaningful auxiliary information. This holds particularly for synthetic aperture radar (SAR) sensors, which use active imaging technology and are thus largely independent from external conditions. In this paper, we investigate whether the inclusion of the radar viewing angle, which is a critical sensor parameter in SAR imaging, provides a benefit for deep learning-based single-image height estimation from VHR SAR data.

This research paper presents a novel approach for fusing time series of cloud-affected optical normalized difference vegetation index (NDVI) data with more regularly sampled synthetic aperture radar (SAR)-estimated NDVI data. We employ a deep learning model based on gated recurrent units (GRU), a recurrent neural network (RNN) cell, that can handle missing data and variable sequence lengths efficiently. Numerical results demonstrate that our method achieves dense and accurate NDVI time series with a mean absolute error (MAE) of 0.05, closely resembling the optical NDVI values. Visual comparison also shows the high performance.

Automated change detection is certainly one of the most discussed applications of remote sensing. However, existing approaches rely on finely co-registered images, as otherwise differences in view point or illumination could lead to erroneous change detections. This is particularly true for very-high-resolution sensors, for which even minor differences might affect several resolution cells. In this paper, we make use of deep learning-based single image height prediction to transform highly non-similar synthetic aperture radar (SAR) images acquired from different viewing angles into homogeneous height maps. The change detection is then carried out in these height maps, thus mitigating any former geometric or radiometric differences of the imagery. An experiment with Capella data observing the city of Mariupol during the Russian war against Ukraine illustrates both the potential and possible risks of the approach.

This paper focuses on the estimation of interferometric SAR parameters, a step that precedes the entire interferometric processing chain to produce derived information such as digital elevation models and ground displacement. Deep learning, especially convolutional neural networks (CNN), has revolutionized image denoising and has recently received considerable attention. However, traditional supervised approaches require labeled images for training, which are generally unavailable or inaccurate, especially in remote sensing applications. To overcome this limitation, semi- and self-supervised denoising approaches have recently been proposed. These can learn from exclusively noisy samples, which can be obtained from pairs of noisy images or from noisy values within the same image. In this paper, we build on the foundation of these self-supervised learning methods, in particular, we borrow concepts from the Noise2Void and Noise2Self approaches, which have already shown excellent performance in various image denoising tasks. We extend this method to address the challenges specific to InSAR phase and coherence estimation, where the complex-valued nature of SAR interferograms poses unique processing considerations.

The presented study investigates the relationship between Near Surface Air temperature (NSAT, Ta) and Thermal Infrared (TIR) measurements obtained from satellites. The investigation utilizes both in-situ and modeled temperature measurements as reference temperatures. Rather than focusing on specific regions, the study adopts a generalized approach by examining the global distribution of the dataset. To conduct a preliminary investigation, the study employs a simple Linear Regression (LR) and a Multi-Layer Perceptron (MLP) approach. The investigation using in-situ temperature measurements and Landsat-8 TIR measurements shows a correlation of 0.74 using the linear regression method and 0.82 using the MLP method. Later, the investigation using modeled temperature measurements and Landsat-8 TIR measurements shows a correlation of 0.83 using the linear regression method and 0.92 using the MLP method. Further, the study presented shows a positive influence of land cover class and climate class on the relationship. Drawing from the results, we anticipate our findings will be valuable in exploring the end-to-end relationship between NSAT and TIR measurements.

Datasets play a key role in developing superior building detection approaches. However, most of the previous work focuses on accurate building masks and scale expansion, while the categories are always missing, which hinders the further analysis of urban development and cultures. Therefore, we propose a benchmark for building detection and fine-grained classification from very high-resolution (VHR) satellite imagery. An extensive annotation is performed for about 0.5 million building instances with 12 fine-grained roof types and individual polygons. The annotation of building functions of two cities in the previous version (UBCv1) [1] is also integrated. To ensure the building variety, it consists of VHR optical images of 20 unique cities worldwide with various landforms and styles of architecture. Its variety and fine-grained categories pose great challenges and meanwhile provide a foundation for the building extraction and fine-grained classification on a global scale. Besides, 17 cities are provided with finely aligned Synthetic Aperture Radar (SAR) images, which can be employed for the development and evaluation of approaches optionally based on optical, SAR, or multi-modal images. Significantly, the proposed benchmark is used as the base of the 2023 IEEE GRSS Data Fusion Contest [2]. The dataset and codes of the baseline methods are available at: https://github.com/AICyberTeam/UBC-dataset/tree/UBCv2.

On average, about half of all optical satellite data observing Earth is covered by haze or clouds. These atmospheric disturbances hinder the ongoing observation of our planet and prevent the seamless application of established remote sensing methods. Accordingly, to allow for an ongoing monitoring of Earth, approaches to reconstruct optical space-borne observations are required. This work introduces a new data set, SEN12MS-CR-TS, for the purpose of multi-sensor time series cloud removal. SEN12MS-CR-TS consists of co-registered radar and optical satellite data, featuring a se-quence of bi-weekly observations throughout an entire year. Finally, we demonstrate the usability of our novel data set by developing a new multi-sensor time-series cloud removal ar-chitecture. We are positive that our curated data set as well as the proposed model will advance future research in satellite image reconstruction and benefit the expanding adaptation of global and all-weather remote sensing applications.

In the era of deep learning, annotated datasets have become a crucial asset to the remote sensing community. In the last decade, a plethora of different datasets was published, each designed for a specific data type and with a specific task or application in mind. In the jungle of remote sensing datasets, it can be hard to keep track of what is available already. With this paper, we introduce EOD - the IEEE GRSS Earth Observation Database (EOD) - an interactive online platform for cataloguing different types of datasets leveraging remote sensing imagery.

This paper introduces the MapInWild dataset, a multi-modal dataset tailored to mapping wilderness areas from satellite imagery and auxiliary geodata. MapInWild accommodates freely and globally available geodata layers that emerged from various remote sensing sensors, such as dualpol Sentinel-1 imagery, multi-spectral Sentinel-2 data, Visible Infrared Imaging Radiometer Suite night-time light data, and the ESA WorldCover map. Each sample of the Map-InWild dataset is annotated with labels derived from the World Database on Protected Areas, a most up-to-date and comprehensive global database on conservation areas. Protected areas are filtered through a sophisticated sampling process to ensure a representative coverage of the natural areas of the Earth. With MapInWild dataset, we hope to foster further research on deep learning applied to environmental remote sensing and conservation. MapInWild dataset is publicly available at https://dataverse.harvard.edu/dataverse/mapinwild.

Height estimation from single images has become a highly requested topic in the remote sensing community. While most methods use optical data, a first attempt with SAR intensity imagery was recently performed and achieved promising results. The actual practical value of any Deep Learning-based method such as this, however, then depends on how well it can be applied to sceneries unknown to the model, maybe even recorded under different acquisition conditions. This paper focuses on that very aspect of this methodology. For this purpose, the differences of distinct data types and test scenes are highlighted and the obtained results by the pre-trained models are evaluated and interpreted.

Monitoring vegetation is of great importance for many applications, for example agriculture or forestry. Commonly, the normalized difference vegetation index (NDVI) of spaceborn sensor is utilized for this task. However, as the NDVI is derived from multispectral optical data, cloud coverage prevents the acquisition of useful values. This results in data and monitoring gaps. Generally, this can be avoided using cloud penetrating radar sensors but the different sensing method and different image characteristics hamper the easy usage of the data. Therefore, in this paper a method is presented to allow global cloud-independent vegetation monitoring by estimating the NDVI from radar data using a deep learning model. The used U-Net architecture is trained on a newly created dataset called SEN12TP of globally distributed radar and optical imagery with a small difference in acquisition time. The resulting performance is evaluated and different input modalities are compared. Additionally, the ability of this approach to densify NDVI time series is demonstrated.

Image-to-image translation between different imaging modalities in Earth observation has become a widely utilized application area of deep learning. However, most of the translation is performed on real-valued data, to some extent neglecting the opportunities of complex-valued SAR data for interferometric methods. In this work, we propose a multi-task deep learning approach for simulating complex-valued InSAR data based on splitting the overall task into multi-modal image-toimage translation sub-tasks. Instead of synthesizing complex-valued SAR data directly, magnitudes, phase values and coherence magnitudes are simulated in parallel and combined to full complex-valued information afterward. With experiments on a Sentinel-1 interferogram, conditioned by DEM and land cover data, we demonstrate the feasibility of the approach.

Extracting buildings from very high-resolution satellite images is a challenging yet important task for applications such as urban monitoring. Multiscale feature learning proves to be a potential solution toward accurate extraction of buildings. This study exploits a powerful multiscale feature learning module, a hierarchical vision transformer by shifted windows (swin), as a backbone within a building extraction network. To this end, we first designed a general structure for building extraction, consisting of a backbone to extract multiscale features and a head network to fuse and refine features. Then, we integrated swin into the structure as a backbone and utilized channel-wise and spatial-wise enhancement in a head network. Experimental results show that our method achieves improvements regarding both F1-score and intersection over union (IoU) compared to the multiple attending path neural network (MAP-Net), which is the current state-of-the-art (SOTA) algorithm for building extraction from remote sensing images. Our study thus confirms the potential of swin transformers as backbones for semantic segmentation tasks based on satellite images.

Human settlement extent (HSE) and local climate zone (LCZ) maps are both essential sources, e.g., for sustainable urban development and Urban Heat Island (UHI) studies. Remote sensing (RS)- and deep learning (DL)-based classification approaches play a significant role by providing the potential for global mapping. However, most of the efforts only focus on one of the two schemes, usually on a specific scale. This leads to unnecessary redundancies since the learned features could be leveraged for both of these related tasks. In this letter, the concept of multitask learning (MTL) is introduced to HSE regression and LCZ classification for the first time. We propose an MTL framework and develop an end-to-end convolutional neural network (CNN), which consists of a backbone network for shared feature learning, attention modules for task-specific feature learning, and a weighting strategy for balancing the two tasks. We additionally propose to exploit HSE predictions as a prior for LCZ classification to enhance the accuracy. The MTL approach was extensively tested with Sentinel-2 data of 13 cities across the world. The results demonstrate that the framework is able to provide a competitive solution for both tasks.

We synthesize both optical RGB and synthetic aperture radar (SAR) remote sensing images from land cover maps and auxiliary raster data using generative adversarial networks (GANs). In remote sensing, many types of data, such as digital elevation models (DEMs) or precipitation maps, are often not reflected in land cover maps but still influence image content or structure. Including such data in the synthesis process increases the quality of the generated images and exerts more control on their characteristics. Spatially adaptive normalization layers fuse both inputs and are applied to a full-blown generator architecture consisting of encoder and decoder to take full advantage of the information content in the auxiliary raster data. Our method successfully synthesizes medium (10 m) and high (1 m) resolution images when trained with the corresponding data set. We show the advantage of data fusion of land cover maps and auxiliary information using mean intersection over unions (mIoUs), pixel accuracy, and Fréchet inception distances (FIDs) using pretrained U-Net segmentation models. Handpicked images exemplify how fusing information avoids ambiguities in the synthesized images. By slightly editing the input, our method can be used to synthesize realistic changes, i.e., raising the water levels. The source code is available athttps://github.com/gbaier/rs_img_synth, and we published the newly created high-resolution data set athttps://ieee-dataport.org/open-access/geonrw.

About half of all optical observations collected via spaceborne satellites are affected by haze or clouds. Consequently, cloud coverage affects the remote-sensing practitioner’s capabilities of a continuous and seamless monitoring of our planet. This work addresses the challenge of optical satellite image reconstruction and cloud removal by proposing a novel multimodal and multitemporal data set called SEN12MS-CR-TS. We propose two models highlighting the benefits and use cases of SEN12MS-CR-TS: First, a multimodal multitemporal 3-D convolution neural network that predicts a cloud-free image from a sequence of cloudy optical and radar images. Second, a sequence-to-sequence translation model that predicts a cloud-free time series from a cloud-covered time series. Both approaches are evaluated experimentally, with their respective models trained and tested on SEN12MS-CR-TS. The conducted experiments highlight the contribution of our data set to the remote-sensing community as well as the benefits of multimodal and multitemporal information to reconstruct noisy information. Our data set is available athttps://patrickTUM.github.io/cloud_removal.

Arc synthetic aperture radar (ArcSAR) is a ground-based remote imaging technology with the ability to cover wide fields of view. However, because its azimuth is formed by scanning angle rotation, the focusing of ArcSAR images is different from classical SAR focusing. Since the existing imaging methods cannot give a good balance between computational efficiency and accuracy, the wavenumber domain algorithm (WMA) could become an interesting alternative. Due to the fact that in ArcSAR imaging, the slant range measurement depends on a sine term of the scanning angle, no dedicated WMA-based approach for ArcSAR imaging has been formulated yet. The main challenge in this context is that the solution of the stationary phase point cannot be resolved explicitly via Fourier transform (FT). This article proposes a new wavenumber domain imaging method, which exploits the sine law in the process of solving the stationary phase point during FT along the direction of the received echo using the triangular relationship formed by the target, the radar, and the rotation center of radar and then obtains the exact phase error expression in range and angular wavenumber domain without any approximation of the slant range or scanning angle. Using this formulation, we develop the corresponding phase error compensation method and complete image focusing. Through point target simulation and experiments on real ArcSAR data, the effectiveness of this method is verified in terms of imaging accuracy and computational efficiency.

Annotated datasets have become one of the most crucial preconditions for the development and evaluation of machine learning-based methods designed for the automated interpretation of remote sensing data. In this paper, we review the historic development of such datasets, discuss their features based on a few selected examples, and address open issues for future developments.

Many observations acquired via optical satellites are polluted by cloud coverage, impeding a continuous and on-demand monitoring of the Earth. Recent advances in the field of cloud removal consider multi-temporal data to reconstruct pixels covered by clouds at a time point of interest. Yet, the limitation of preceding work is that information gets integrated over time, removing any temporal resolution from the de-clouded end products. In this work we consider a sequence-to-sequence approach, translating cloudy time series to a series of cloud-free multi-spectral images without the need of any external cloud-free data set. Our network is guided by synthetic aperture radar (SAR) information providing a strong prior for the reconstruction of cloud-covered information. We analyze the proposed method by visual inspection of predictions and in terms of error metrics to highlight its benefits. Finally, an ablation study is conducted in which the our network is compared against a baseline model and the effectiveness of the proposed SAR prior is demonstrated.

The automatic matching of corresponding pixels in SAR and optical remote sensing imagery has been an active field of research for many years. While early approaches were usually based on the measurement of image similarity by signal-based measures or hand-crafted image features, more recent matching techniques make use of deep learning. Since the different approaches proposed in the literature are usually trained and evaluated on specific, individual datasets, i.e. with unique input data and target label criteria, a direct comparison has not yet been possible. With this paper, we intend to close that gap by providing the first comparative evaluation of different state-of-the-art deep learning-based SAR-optical image matching approaches.

Decorrelation phenomena are always present in synthetic aperture radar interferometry (InSAR). While this implies a certain level of signal degradation, decorrelation is also a characteristic of the type of imaged target itself and can, therefore, be seen as a source of information. In this letter, we investigate InSAR decorrelation effects at the X-band by fitting volume and temporal decorrelation trends using the unique combination of data provided by the TanDEM-X (TDX) and PAZ spaceborne missions. The innovative use of this constellation allows for the acquisition of both single- and repeat-pass data at short revisit times. The concurrent availability of simultaneous acquisitions and the fine temporal resolution makes this constellation the ideal observation scenario for the study of decorrelation phenomena. Overall, we analyze five test sites, characterized by the presence of different land cover classes, and for each of them, we provide volume and temporal decorrelation fitting parameters. The performed analysis gives a first insight on the potential of combining bistatic and repeat-pass InSAR acquisitions also in view of future spaceborne constellations, which could benefit from the TDX/PAZ experience.

The majority of optical observations acquired via spaceborne Earth imagery are affected by clouds. While there is numerous prior work on reconstructing cloud-covered information, previous studies are, oftentimes, confined to narrowly defined regions of interest, raising the question of whether an approach can generalize to a diverse set of observations acquired at variable cloud coverage or in different regions and seasons. We target the challenge of generalization by curating a large novel data set for training new cloud removal approaches and evaluate two recently proposed performance metrics of image quality and diversity. Our data set is the first publically available to contain a global sample of coregistered radar and optical observations, cloudy and cloud-free. Based on the observation that cloud coverage varies widely between clear skies and absolute coverage, we propose a novel model that can deal with either extreme and evaluate its performance on our proposed data set. Finally, we demonstrate the superiority of training models on real over synthetic data, underlining the need for a carefully curated data set of real observations. To facilitate future research, our data set is made available online.

The majority of optical images acquired via spaceborne remote sensing are affected by clouds. Recent advances in cloud removal combine multimodal data with deep neural networks recovering the affected areas. To relax the requirements on the data the network is trained on previous approaches utilized generative models no longer necessitating strict pixel-wise correspondences between cloudy input and cloud-free target images. However, such models are often-times prone to fiction, i.e. the generation of content systematically differing from the structure of the target images. In this work we combine the fusion of optical and radar imagery with the advantages of generative models trainable on unpaired optical data, while reducing fiction by reconstructing optical information only where it need be-over cloud-covered areas. We evaluate our approach qualitatively and quantitatively and demonstrate its effectiveness.

Local Climate Zone (LCZ) classification is the most commonly used scheme to analyze how local urban morphology affects the climate of local areas. Classification methods are often based on remote sensing data or on a fusion of several data sources. In this study, the effects of different fusion strategies of optical and synthetic aperture radar (SAR) data on the accuracy of LCZ classifications are investigated. The data processing is implemented with a convolutional neural network (CNN), where until a fusion layer, separate data sources are processed separately on branches. Strategies of splitting the data into branches and the effects of different fusion stages are compared, together with approaches based on sums of independent classifiers. For our setting, the stage of fusion does not seem to have a big influence on the accuracy. The results of this study contribute to a better understanding of cooperative usage of multispectral and SAR data.

Exploiting multitemporal Sentinel-2 images for urban land cover classification has become an important research topic, since these images have become globally available at relatively fine temporal resolution, thus offering great potential for large-scale land cover mapping. However, appropriate exploitation of the images needs to address problems such as cloud cover inherent to optical satellite imagery. To this end, we propose a simple yet effective decision-level fusion approach for urban land cover prediction from multiseasonal Sentinel-2 images, using the state-of-the-art residual convolutional neural networks (ResNet). We extensively tested the approach in a cross-validation manner over a seven-city study area in central Europe. Both quantitative and qualitative results demonstrated the superior performance of the proposed fusion approach over several baseline approaches, including observation- and feature-level fusion.

The automatic matching of corresponding regions in remote sensing imagery acquired by synthetic aperture radar (SAR) and optical sensors is a crucial pre-requesite for many data fusion endeavours such as target recognition, image registration, or 3D-reconstruction by stereogrammetry. Driven by the success of deep learning in conventional optical image matching, we have carried out extensive research with regard to deep matching for SAR-optical multi-sensor image pairs in the recent past. In this paper, we summarize the achieved findings, including different concepts based on (pseudo-)siamese convolutional neural network architectures, hard negative mining, alternative formulations of the underlying loss function, and creation of artificial images by generative adversarial networks. Based on data from state-of-the-art remote sensing missions such as TerraSAR-X, Prism, Worldview-2, and Sentinel-1/2, we show what is already possible today, while highlighting challenges to be tackled by future research endeavors.

This paper proposes a framework to fuse multi-seasonal Sentinel-2 images, with application on LCZ-derived urban land cover classification. Cross-validation over a seven-city study area in central Europe demonstrates its consistently better performance over several previous approaches, with the same experimental setup. Based on our previous work, we can conclude that decision-level fusion is better than feature-level fusion for similar tasks at similar scale with multi-seasonal Sentinel-2 images. With the framework, urban land cover maps of several cities are produced. The visualization of two exemplary areas shows urban structures that are consistent with existing datasets. This framework can be also generally beneficial for other types of urban mapping.

The fusion of heterogeneous image data, in particular optical images and synthetic aperture radar (SAR) images, is highly worthwhile in the context of remote sensing tasks as it allows to exploit complementary information - such as spectral and distance measurements or different observation perspectives - of the two data sources while diminishing their individual weaknesses (e.g. cloud cover, difficulty of image interpretation, limited sensor revisit). However, relating the heterogeneous data on the signal level requires a data alignment step, which cannot be realized without auxiliary knowledge. This paper addresses and discusses this fundamental fusion problem in remote sensing in the context of a framework named SimGeoI, which solves the multi-sensor alignment task based on geometric knowledge from existing digital surface models. Sections of optical and SAR images are related to individual objects using interpretation layers generated with ray tracing techniques. Results of SimGeoI are presented for a test site in London in order to motivate an object-related fusion of remote sensing images.

Recently, the TanDEM-X DEM has been produced as a global DEM with unprecedented relative accuracy. One important step of the chain of global DEM generation is to mosaic multiple raw DEM tiles by DEM fusion methods to reach the best possible target accuracy. Currently, Weighted Averaging (WA) is used as a fast and simple method for TanDEM-X raw DEM fusion in which the weights are computed from height error maps delivered from the Interferometric TanDEM-X Processor (ITP). In this paper, we investigate the efficiency of variational models such as TV-L1 and Huber model for the TanDEM-X raw DEM fusion task in comparison to WA. The results illustrate that using variational models can improve the quality of DEM fusion outputs especially for areas with high-frequency contents and more complex morphological features like urban areas. Using variational models could improve the DEM quality by up to about 1m.

In this paper, we present the first conditional generative adversarial network (cGAN) architecture that is specifically designed to fuse synthetic aperture radar (SAR) and optical multi-spectral (MS) image data to generate cloud- and haze-free MS optical data from a cloud-corrupted MS input and an auxiliary SAR image. Experiments on Sentinel-2 MS and Sentinel-l SAR data confirm that our extended SAR-Opt-cGAN model utilizes the auxiliary SAR information to better reconstruct MS images than an equivalent model which uses the same architecture but only single-sensor MS data as input.

In this paper we propose a deep generative framework, based on a generative adversarial network (GAN) and an auto encoder (AE), for generating non-corresponding SAR patches to be used in hard negative mining in situations of limited data quantities. We evaluate the effectiveness of this formulation of hard negative mining for reducing the false positive rate (FPR) and improving network determinability in a SAR-optical patch matching application. Our generative network is trained to generate realistic SAR images using an existing SAR-optical matching dataset. These generated images are then used as non-corresponding, hard negative samples for training a SAR-optical matching network. Our results show that we are able to generate realistic SAR images which exhibit many SAR-like features, such as layover and speckle. We further show that by fine tuning the original matching network using these hard negative samples we are able to improve the overall performance of the original SAR-optical matching network.

This paper evaluates different spectral-spatial features that can be extracted from Sentinel-2 imagery regarding their relevance for discriminating different Local Climate Zone (LCZ) classes. The features include spectral reflectance, spectral indices, Morphological Profiles (MPs), as well as Global Urban Footprint (GUF), the Open Street Map layers buildings and land use, and their combinations. Using a residual convolutional neural network (ResNet), a systematic analysis of feature importance is performed with a manually generated dataset distributed in Europe. The results of this evaluation are meant to provide guidance about the choice of both spectral and spatial features for the task of LCZ classification on a global scale. The results show that GUF and OSM can contribute to the classification performance, and ResNet relies less on additional features with the highest accuracy provided by the reflectance only.

In this letter, we propose a pseudo-siamese convolutional neural network architecture that enables to solve the task of identifying corresponding patches in very high-resolution optical and synthetic aperture radar (SAR) remote sensing imagery. Using eight convolutional layers each in two parallel network streams, a fully connected layer for the fusion of the features learned in each stream, and a loss function based on binary cross entropy, we achieve a one-hot indication if two patches correspond or not. The network is trained and tested on an automatically generated data set that is based on a deterministic alignment of SAR and optical imagery via previously reconstructed and subsequently coregistered 3-D point clouds. The satellite images, from which the patches comprising our data set are extracted, show a complex urban scene containing many elevated objects (i.e., buildings), thus providing one of the most difficult experimental environments. The achieved results show that the network is able to predict corresponding patches with high accuracy, thus indicating great potential for further development toward a generalized multisensor key-point matching procedure.

The most unique advantage of multipass synthetic aperture radar interferometry (InSAR) is the retrieval of long-term geophysical parameters, e.g., linear deformation rates, over large areas. Recently, an object-based multipass InSAR framework has been proposed by Kang, as an alternative to the typical single-pixel methods, e.g., persistent scatterer interferometry (PSI), or pixel-cluster-based methods, e.g., SqueeSAR. This enables the exploitation of inherent properties of InSAR phase stacks on an object level. As a follow-on, this paper investigates the inherent low rank property of such phase tensors and proposes a Robust Multipass InSAR technique via Object-based low rank tensor decomposition. We demonstrate that the filtered InSAR phase stacks can improve the accuracy of geophysical parameters estimated via conventional multipass InSAR techniques, e.g., PSI, by a factor of 10-30 in typical settings. The proposed method is particularly effective against outliers, such as pixels with unmodeled phases. These merits, in turn, can effectively reduce the number of images required for a reliable estimation. The promising performance of the proposed method is demonstrated using high-resolution TerraSAR-X image stacks.

In this paper, we summarize challenges, proposed solutions and recent trends in the field of SAR-optical remote sensing data fusion. Although being a pre-processing step before the actual fusion-by-estimation, it is shown that matching and coregistration is one of the core challenges in that regard, which is mainly due to the strongly different geometric and radiometric properties of the two observation types. We then review some of the published fusion methods and discuss the future trends of this topic.

This paper presents the basics and functionality of SimGeoI, a simulation-based framework for the automated interpretation and alignment of optical and SAR remote sensing data. SimGeoI has been developed in order to align optical and SAR data based on given geometric information about objects represented by digital surface models. Thereby, the analysis of urban scenes is possible with independence of sensor type and perspective. After a brief introduction of the processor environment, possible applications of the framework are indicated with results of a case study for Istanbul (WorldView-2 and TerraSAR-X data). In this context, opportunities in the context of a joint analysis of high resolution optical and SAR data are addressed, i.e. concerning data fusion, change detection, and machine learning tasks.

This paper deals with TanDEM-X and Cartosat-1 DEM fusion over urban areas with support of weight maps predicted by an artificial neural network (ANN). Although the TanDEM-X DEM is a global elevation dataset of unprecedented accuracy (following HRTI-3 standard), its quality decreases over urban areas because of artifacts intrinsic to the SAR imaging geometry. DEM fusion techniques can be used to improve the TanDEM-X DEM in problematic areas. In this investigation, Cartosat-1 elevation data were fused with the TanDEM-X DEM by weighted averaging and total variation (TV)-based regularization, resorting to weight maps derived by a specifically trained ANN. The results show that the proposed fusion strategy can significantly improve the final DEM quality.

In this paper, we investigate making use of a convolutional neural network (CNN) to solve the task of identifying corresponding patches in very high resolution (VHR) optical and SAR imagery of complicated urban scenery. By doing so, the binary decision function is learnt directly from automatically generated training data and does not resort to any hand-crafted features. First evaluations show great potential for further studies towards a generalized multi-sensor matching procedure.

This paper compares different similarity measures for the matching of very-high-resolution SAR and optical images over urban areas. It is meant to provide guidance about the performance of both signal-based and descriptor-based similarity measures in the context of this non-trivial case of multi-sensor correspondence matching. Using an automatically generated training dataset, thresholds for the distinction between correct matches and wrong matches are determined. It is shown that descriptor-based similarity measures outperform signal-based similarity measures significantly.

This paper presented experimental results discussing the potential of millimeterwave SAR for forest remote sensing on the individual tree level. As can be seen from the experimental results, although there is a certain amount of canopy penetration, a significant part of the signal response is received from the tree crowns. This provides both interesting perspectives for an analysis of forest volumes by continuous TomoSAR models as well as the reconstruction of individual tree models by utilization of discrete TomoSAR models.

In this letter, for the first time, the potential of millimeterwave synthetic aperture radar (SAR) is investigated with respect to a tomographic analysis of forest volumes. Exploiting both parametric and nonparametric SAR tomography (TomoSAR) methods designed for both discrete and continuous reflectivity profiles, it is shown that even Ka-band signals with a wavelength of only 8.55 mm can penetrate the tree canopy to a certain extent and allow a separation of ground and tree crowns. First experimental results exploiting airborne multiantenna data are evaluated with respect to LiDAR ground truth and indicate a promising perspective.

The detection of coastlines in SAR imagery has been studied for more than two decades now. Whereas the first works were based on the exploitation of amplitude imagery and the corresponding need to deal with speckle noise [1, 2, 3], with the ERS-1/2 tandem configuration also coherence maps started to be used as input. Based on the insights gained on these experiments, later the authors began to exploit both amplitude and coherence imagery simultaneously, finally giving way to the first approach using the original complex SAR data for statistically motivated coastline extraction.

Automatic recognition of single trees in remote sensing data is an important research topic in the context of sustainable forest management: In many countries, single-tree related parameters are used as a basis for forest inventory, e.g. tree species, mean tree height or timber volume. Until now, the majority of these parameters are collected manually by measurement of sample plots in cost- and time-intensive field surveys. However, remote sensing-based methods have gained increasing attention during recent decades [1]. In the remote sensing context, an additional application of single tree extraction is driven by the goal to generate information for city tree cadastres or to add layers to geoinformation systems and 3D city models [2].

Multilooking is a critical task in interferometric synthetic aperture radar (SAR) imaging. While there are many algorithms designed for SAR image pairs and also some first approaches for multi-temporal satellite data stacks, no method suitable to airborne single-pass stacks that typically contain just a small number of multi-baseline acquisitions has been proposed yet. This paper presents an adaptive procedure to determine regions of homogeneous backscattering in heterogeneous scenes such as urban areas. Based on these regions, the complex covariance matrices can be estimated for all pixels in the stack. This step enables the retrieval of all relevant information of the multi-baseline InSAR data set, e.g., despeckled intensity images, interferometric phase observations, and related coherence maps. The denoising efficiency of the proposed method is evaluated and compared to different algorithms. Furthermore, the detail preservation is analyzed in order to prove the validity of the homogeneity assumption.

For many multidimensional applications of synthetic aperture radar (SAR) imaging, the estimation of the covariance matrix for each resolution cell is a critical processing step. The context of this work is the application of covariance matrix estimation for multi-baseline interferometric SAR data sets. In order to ensure local stationarity, which is needed for an unbiased estimation, adaptive techniques are necessary. In this paper, a new approach for adaptive covariance matrix estimation is proposed and evaluated based on measures known from the field of image processing. The procedure is centered around the idea of checking whether the neighboring pixels belong to the same statistical distribution as the currently investigated pixel by applying a threshold to the respective probability density function. All inlier pixels are then used to estimate the complex covariance matrix of the reference pixel. From this covariance matrix, both amplitude and interferometric phase values are extracted, which are then combined for all pixels in the stack in order to employ techniques for the evaluation of filtering efficiency that are typically used in image denoising research. It is found that the proposed algorithm provides high filtering efficiency and good detail preservation at the same time. Apart from that, it is found to be particularly suitable for small-sized stacks of coregistered SAR imagery.

In this letter, compressive sensing based methods for layover separation in airborne single-pass multi-baseline InSAR data are investigated. The standard compressive sensing (CS) is compared with recently proposed distributed CS (DCS) and a new “multilooking” approach to CS (MCS). Experiments on simulated data show that, while CS is not satisfyingly applicable to single-pass data stacks with just few acquisitions, the neighborhood-based approaches, DCS and MCS, yield a promising perspective.

In this paper, we discuss invalidity of the linear mixture model of independent component analysis (ICA) and evaluate an estimation method for the polarization orientation (PO) angle in order to correct the mismatched data model regarding unmixing signals of polarimetric SAR imagery. Firstly, the linear mixture model of ICA and the nonlinear mixture model of the PO effect are described so that we refer to distorted assumption. Then, a correction method for the PO angle is introduced and tested with single- and multi-temporal L-band images in two areas. We employ two indices of class separation and statistical independence for evaluation since they are supported concerning fundamental theory and the scale problem of ICA. They show its performance of feature extraction and recovery from the invalid mixture model. With these results, we show the effectiveness of nonlinear correction for the application of ICA to polarimetric SAR imagery.

Hysteresis smoothing is a filtering method for one-dimensional data, which removes waveform noise with a simple but powerful process. Although it may be applied in two-dimensional data, an inherent problem of severe artifacts makes it difficult to use this method on SAR imagery. In this paper, we take notice of a property appeared when hysteresis smoothing works well on homogeneous areas. We propose a simple rule to exploit this advantage without losing the merits of the original approach. The effectiveness of our approach is demonstrated quantitatively and qualitatively with simulated and real Ku-band SAR data.

In this work we investigate the automatic detection of stationary vehicles in SAR images by supervised learning algorithms. This implies the description of the vehicles by a set of representative features. We combine several classes of features including subspace projection based on clustering mechanisms (NMF, PCA), statistical features (image moments), spectral features (gabor wavelets) as well as boundary (shape analysis) and region descriptors (HOG). We further use two different learning algorithms: Support Vector Machines (SVM) and Random Forests.

Multilooking is one of the most important processing steps in SAR interferometry. While formerly fixed-size box-car windows have been used, the problem has become less trivial since decimeter resolution sensors have made a detailed analysis of urban areas possible. This paper presents an approach to detect neighborhoods of homogeneous backscattering in airborne multi-baseline InSAR data stacks. Based on these neighborhoods, an adaptive estimation of the covariance matrix as well as multilooking of interferometric phase and coherence become possible.

This paper addresses the generation of “true” RADAR ortho-images from highest resolution multi-aspect InSAR data. Due to the side-looking SAR imaging geometry, the well-known layover and shadowing effects prevent the production of truly rectified ortho-imagery from one image alone. Here, an approach for the reconstruction of Digital Surface Models of densely built inner city areas is proposed. Since in SAR interferometry each dataset pixel contains not only the interferometric phase needed for 3D reconstruction but also the corresponding amplitude or intensity value, respectively, the procedure can also be seen in the context of true ortho-rectification.