As large-scale foundation models become publicly available for different domains, efficiently adapting them to individual downstream applications and additional data modalities has turned into a central challenge. For example, foun-dation models for geospatial and satellite remote sensing applications are commonly trained on large optical RGB or multi-spectral datasets, although data from a wide variety of heterogeneous sensors are available in the remote sensing domain. This leads to significant discrepancies between pre-training and downstream target data distributions for many important applications. Fine-tuning large foundation models to bridge that gap incurs high computational cost and can be infeasible when target datasets are small. In this paper, we address the question of how large, pre-trained foundational transformer models can be efficiently adapted to downstream remote sensing tasks involving different data modalities or limited dataset size. We present a self-supervised adaptation method that boosts downstream linear evaluation accuracy of different foundation models by 4-6% (absolute) across 8 remote sensing datasets while outperforming full fine-tuning when training only 1-2% of the model parameters. Our method significantly improves label efficiency and increases few-shot accuracy by 6-10% on different datasets11Code available at github. com/HSG-AIML/GDA .

The analysis of soil organic carbon (SOC) contents and stocks provides crucial insights into soil health, biodiversity support, erosion control, and climate change mitigation. To enhance SOC level estimation and reveal patterns across diverse ecosystems, advanced deep learning techniques are integrated with remote sensing data. Despite this progress, designing a powerful network capable of handling unevenly distributed ground truths remains a challenge, reflecting the inherent complexity of the data in such scenarios. In this study, we propose a hybrid deep learning network that incorporates a vision transformer (ViT) for processing remote sensing images and a transformer for handling climate features. To address the challenges posed by particularly difficult-to-predict samples, we apply a cost-sensitive loss function within this network. Experimental results on the LUCAS dataset demonstrate that this new loss function significantly outperforms conventional ones.1

Self-supervised pretraining has been shown to greatly improve the performance and label-efficiency of Deep Learning models for multi-modal remote sensing applications. In this work, we explore the use of diffusion methods in a multi-modal setup as a means to pretrain architectures in a task-agnostic way. We qualitatively find that multi-modal diffusion processes are able to coherently learn information across multiple data-modalities. Fine-tuning the pretrained backbone on a segmentation task outperforms a supervised baseline model and is highly label-efficient, which is not the case when fine-tuned on a classification task.

Soil organic carbon (SOC) constitutes a fundamental component of terrestrial ecosystem functionality, playing a pivotal role in nutrient cycling, hydrological balance, and erosion mitigation. Precise mapping of SOC distribution is imperative for the quantification of ecosystem services, notably carbon sequestration and soil fertility enhancement. Digital soil mapping (DSM) leverages statistical models and advanced technologies, including machine learning (ML), to accurately map soil properties, such as SOC, utilizing diverse data sources like satellite imagery, topography, remote sensing indices, and climate series. Within the domain of ML, self-supervised learning (SSL), which exploits unlabeled data, has gained prominence in recent years. This study introduces a novel approach that aims to learn the geographical link between multimodal features via self-supervised contrastive learning, employing pretrained Vision Transformers (ViT) for image inputs and Transformers for climate data, before fine-tuning the model with ground reference samples. The proposed approach has undergone rigorous testing on two distinct large-scale datasets, with results indicating its superiority over traditional supervised learning models, which depends solely on labeled data. Furthermore, through the utilization of various evaluation metrics (e.g., root-mean-square error (RMSE), mean absolute error (MAE), concordance correlation coefficient (CCC), etc.), the proposed model exhibits higher accuracy when compared to other conventional ML algorithms like random forest and gradient boosting. This model is a robust tool for predicting SOC and contributes to the advancement of DSM techniques, thereby facilitating land management and decision-making processes based on accurate information.

In supervised learning, which is commonly used in Remote Sensing applications, the performance of a model trained on a larger dataset is generally better than or equal to a model trained on a smaller dataset. Dataset distillation is a method that extracts the discriminative features from a larger dataset to a smaller one. By doing so, the important characteristics of the original dataset that are critical for learning can be isolated. This has implications regarding computational efficiency and understanding underlying representation learning dynamics. These implications are of particular interest in the context of remote sensing, where large amounts of data are being generated and processed every day. We use dataset distillation across multiple network architectures on the RGB bands of the EuroSAT dataset to test how it would behave in a Remote Sensing scenario with real-world data. Our distilled dataset leads to a consistent out-performance of 5%-10% compared to random sampling for downstream classification tasks.

Deep learning methods have proven to be a powerful tool in the analysis of large amounts of complex Earth observation data. However, while Earth observation data are multi-modal in most cases, only single or few modalities are typically considered. In this work, we present the ben-ge dataset, which supplements the BigEarthNet-MM dataset by compiling freely and globally available geographical and environmental data. Based on this dataset, we showcase the value of combining different data modalities for the downstream tasks of patch-based land-use/land-cover classification and land-use/land-cover segmentation. ben-ge is freely available and expected to serve as a test bed for fully supervised and self-supervised Earth observation applications.

Fossil fuel combustion produces large quantities of carbon dioxide (CO2), a major greenhouse gas (GHG), which is one of the main drivers of climate change. A quantitative assessment of GHG emissions is fundamental to predicting climate change effects, enforcing emission regulations, and monitoring pollution trading schemes. Unfortunately, the reporting of GHG emissions is only required in some countries, resulting in insufficient global coverage. At the same time, the transition from fossil fuels to zero carbon to limit climate change is at the heart of several ecological movements, hence the need for quantifying energy production, as well. In this work, we propose an end-to-end method to estimate power generation rates for fossil fuel power plants from satellite images, based on which we approximate GHG (CO2) emission rates. We present a physics-guided multitask deep-learning approach able to simultaneously predict from a single-satellite image of a power plant: 1) the pixel-area covered by plumes; 2) the type of fired fuel; and 3) the power generation rate. To ensure physically realistic predictions from our model we account for environmental conditions and empirical physical constraints. We then convert the predicted power generation rate into estimates for the rate at which CO2is being emitted, using a fuel-dependent conversion factor. Experimental results show that our multitask learning approach improves the power generation estimation mean absolute error (MAE) by 23% compared to a single-task network trained on the same dataset.

Road traffic noise represents a global health issue. Despite its importance, noise data are unavailable in many regions of the world. We therefore propose to approximate noise data from satellite imagery in an end-to-end Deep Learning approach. We train a U-Net segmentation model to estimate road noise based on freely available Sentinel-2 satellite imagery and existing road traffic noise estimates for Switzer-land. We are able to achieve an RMSE of 8.8 dB(A) for day-time traffic noise and 7.6 dB(A) for nighttime traffic noise with a spatial resolution of 10 m. In addition to identifying major road networks, our model succeeds to predict the spatial propagation of noise. Our results suggest that this approach provides a pathway to estimating road traffic noise for areas for which no such measures are available.

Satellites allow spatially precise monitoring of the Earth, but provide only limited information on events of societal impact. Subjective societal impact, however, may be quantified at a high frequency by monitoring social media data. In this work, we propose a multi-modal data fusion framework to accurately identify periods of COVID-19-related lockdown in the United Kingdom using satellite observations (NO2measurements from Sentinel-5P) and social media (textual content of tweets from Twitter) data. We show that the data fusion of the two modalities improves the event detection accuracy on a national level and for large cities such as London.

Air pollution is a central environmental problem in countries around the world. It contributes to climate change through the emission of greenhouse gases, and adversely impacts the health of billions of people. Despite its importance, detailed information about the spatial and temporal distribution of pollutants is complex to obtain. Ground-level monitoring stations are sparse, and approaches for modeling air pollution rely on extensive datasets which are unavailable for many locations. We introduce three techniques for the estimation of air pollution to overcome these limitations: 1) a baseline localized approach that mimics conventional land-use regression through gradient boosting; 2) an OpenStreetMap (OSM) approach with gradient boosting that is applicable beyond regions covered by detailed geographic datasets; and 3) a remote sensing-based deep learning method utilizing multiband imagery and trace-gas column density measurements from satellites. We focus on the estimation of nitrogen dioxide (NO2), a common anthropogenic air pollutant with adverse effects on the environment and human health. Our local baseline model achieves strong results with a mean absolute error (MAE) of 5.18 ±$0.16~\mu \text {g/m}^{3}$NO2. Substituting localized inputs with OSM leads to a degraded performance (MAE 7.22 ± 0.14) but enables NO2estimation at a global scale. The proposed deep learning model on remote sensing data combines high accuracy (MAE 5.5 ± 0.14) with global coverage and heteroscedastic uncertainty quantification. Our results enable the estimation of surface-level NO2pollution with high spatial resolution for any location on Earth. We illustrate this capability with an out-of-distribution test set on the US westcoast. Code1and data2are publicly available.

Satellite remote imaging enables the detailed study of land use patterns on a global scale. We investigate the possibility to improve the information content of traditional land use classification by identifying the nature of industrial sites from medium-resolution remote sensing images. In this work, we focus on classifying different types of power plants from Sentinel-2 imaging data. Using a ResNet-50 deep learning model, we are able to achieve a mean accuracy of 90.0% in distinguishing 10 different power plant types and a background class. Furthermore, we are able to identify the cooling mechanisms utilized in thermal power plants with a mean accuracy of 87.5%. Our results enable us to qualitatively investigate the energy mix from Sentinel-2 imaging data, and prove the feasibility to classify industrial sites on a global scale from freely available satellite imagery.

NO2is an atmospheric trace gas that contributes to global warming as a precursor of greenhouse gases and has adverse effects on human health. Surface NO2concentrations are commonly measured through strictly localized networks of air quality stations on the ground. This work presents a novel dataset of surface NO2measurements aligned with atmospheric column densities from Sentinel-5P, as well as geographic and meteorological variables and lockdown information11Available at https://github.com/HSG-AIML/NO2-dataset. The dataset provides access to data from a variety of sources through a common format and will foster data-driven research into the causes and effects of NO2pollution. We showcase the value of the new dataset on the task of surface NO2estimation with gradient boosting. The resulting models enable daily estimates and confident identification of EU NO2exposure limit breaches. Additionally, we investigate the influence of COVID-19 lockdowns on air quality in Europe and find a significant decrease in NO2levels.