Weakly Supervised Semantic Segmentation (WSSS) is a challenging problem that has been extensively studied in recent years. Traditional approaches often rely on external modules like Class Activation Maps to highlight regions of interest and generate pseudo segmentation masks. In this work, we propose an end-to-end method that directly utilizes the attention maps learned by a Vision Transformer (ViT) for WSSS. We propose training a sparse ViT with multiple [CLS] tokens (one for each class), using a random masking strategy to promote [CLS] token - class assignment. At inference time, we aggregate the different self-attention maps of each [CLS] token corresponding to the predicted labels to generate pseudo segmentation masks. Our proposed approach enhances the interpretability of self-attention maps and ensures accurate class assignments. Extensive experiments on two standard benchmarks and three specialized datasets demonstrate that our method generates accurate pseudo-masks, outperforming related works. Those pseudo-masks can be used to train a segmentation model which achieves results comparable to fully-supervised models, significantly reducing the need for fine-grained labeled data.

Deep learning methods have proven to be a powerful tool in the analysis of large amounts of complex Earth observation data. However, while Earth observation data are multi-modal in most cases, only single or few modalities are typically considered. In this work, we present the ben-ge dataset, which supplements the BigEarthNet-MM dataset by compiling freely and globally available geographical and environmental data. Based on this dataset, we showcase the value of combining different data modalities for the downstream tasks of patch-based land-use/land-cover classification and land-use/land-cover segmentation. ben-ge is freely available and expected to serve as a test bed for fully supervised and self-supervised Earth observation applications.

Fossil fuel combustion produces large quantities of carbon dioxide (CO2), a major greenhouse gas (GHG), which is one of the main drivers of climate change. A quantitative assessment of GHG emissions is fundamental to predicting climate change effects, enforcing emission regulations, and monitoring pollution trading schemes. Unfortunately, the reporting of GHG emissions is only required in some countries, resulting in insufficient global coverage. At the same time, the transition from fossil fuels to zero carbon to limit climate change is at the heart of several ecological movements, hence the need for quantifying energy production, as well. In this work, we propose an end-to-end method to estimate power generation rates for fossil fuel power plants from satellite images, based on which we approximate GHG (CO2) emission rates. We present a physics-guided multitask deep-learning approach able to simultaneously predict from a single-satellite image of a power plant: 1) the pixel-area covered by plumes; 2) the type of fired fuel; and 3) the power generation rate. To ensure physically realistic predictions from our model we account for environmental conditions and empirical physical constraints. We then convert the predicted power generation rate into estimates for the rate at which CO2is being emitted, using a fuel-dependent conversion factor. Experimental results show that our multitask learning approach improves the power generation estimation mean absolute error (MAE) by 23% compared to a single-task network trained on the same dataset.

Satellites allow spatially precise monitoring of the Earth, but provide only limited information on events of societal impact. Subjective societal impact, however, may be quantified at a high frequency by monitoring social media data. In this work, we propose a multi-modal data fusion framework to accurately identify periods of COVID-19-related lockdown in the United Kingdom using satellite observations (NO2measurements from Sentinel-5P) and social media (textual content of tweets from Twitter) data. We show that the data fusion of the two modalities improves the event detection accuracy on a national level and for large cities such as London.

Satellite remote imaging enables the detailed study of land use patterns on a global scale. We investigate the possibility to improve the information content of traditional land use classification by identifying the nature of industrial sites from medium-resolution remote sensing images. In this work, we focus on classifying different types of power plants from Sentinel-2 imaging data. Using a ResNet-50 deep learning model, we are able to achieve a mean accuracy of 90.0% in distinguishing 10 different power plant types and a background class. Furthermore, we are able to identify the cooling mechanisms utilized in thermal power plants with a mean accuracy of 87.5%. Our results enable us to qualitatively investigate the energy mix from Sentinel-2 imaging data, and prove the feasibility to classify industrial sites on a global scale from freely available satellite imagery.