Recent self-supervised methods for learning image representations on natural images have shown different approaches to tackle either classification or dense prediction tasks. By utilizing global features per image and a set of local features, they try to tackle the trade-off between these downstream tasks. Unfortunately, remote sensing imagery is seldom comprised of only one central object, but of many different regions with varying types of region boundaries. Thus, many prominent approaches for natural images do not model this inside their local feature learning schemes. We utilize the benefits of local region masks as a prior to representation learning through local feature matching. On dense remote sensing tasks, like semantic segmentation for land-cover mapping our approach yields better results than the current matching schemes and is on par given the trade-off between local and global features for the largest current remote sensing benchmarks with multiple tasks.

Training current deep neural network architectures for automated earth remote sensing raises the necessity of huge amounts of labeled data or the application of suitable transfer learning approaches. Unfortunately, transfer learning representations learned on ImageNet are not well suited for the remote sensing domain. Fortunately, self-supervised learning (SSL) is stepping in to fill this gap, by providing domain specific representations without the need for labels. Moreover, in contrast to natural scenes, earth’s land cover classes change with seasons, which has to be considered during training and data collection. Luckily, unlabeled observations over different seasons at the same location are easy to come by, with a lot of publicly available data from, e.g., the Sentinel-2 mission and automated sampling approaches.

Research on the use of augmentations in physically constrained remote sensing scenarios, like the analysis of Martian surface data, is largely unexplored. In this work we present an analysis on how reasonable augmentation strategies can be selected which are class agnostic and respect physical plausibility in supervised and weakly-supervised tasks. Additionally, we present the first results of self-supervised learning on Martian surface data, discuss the importance of physically plausible augmentations in the context of self-supervised learning, specifically contrastive learning, and provide a comprehensive overview of the generalization properties induced by different augmentation strategies with the help of geomorphic maps.

This work presents SeasoNet, a new large-scale multi-label land cover and land use scene understanding dataset. It includes 1759830 images from Sentinel-2 tiles, with 12 spectral bands and patch sizes of up to$120 \text{px}\times 120{\text{px}}$. Each image is annotated with large scale pixel level labels from the german land cover model LBM-DE2018 with land cover classes based on the CORINE Land Cover database (CLC) 2018 and a five times smaller minimum mapping unit (MMU) than the original CLC maps. We provide pixel synchronous examples from all four seasons, plus an additional snowy set. These properties make SeasoNet the currently most versatile and biggest remote sensing scene understanding dataset with possible applications ranging from scene classification over land cover mapping to content-based cross season image retrieval and self-supervised feature learning. We provide base-line results by evaluating state-of-the-art deep networks on the new dataset in scene classification and semantic segmentation scenarios.

In this work we aim to bridge remote sensing scene understanding and mapping on Earth and Mars. Accordingly, we present common challenges to both domains, like an abundance of data, class imbalances, and the scarcity of high resolution ground-truth data. Additionally, we introduce a novel multi-modal semantic segmentation dataset which is based on a geologic map of a rover landing site on Mars and distill the aforementioned challenges into a single dataset. We present initial results and discuss how well a deep neural network can learn to create geologic maps from orbital images of Mars.

Through the rise of new remote sensing datasets with sufficient size for current deep learning models, land cover classification results have improved in recent years. Unfortunately, earth exhibits a natural imbalance in different land cover classes, also visible in these datasets. In the domain of zero-shot learning, image attributes have enabled better results in transfer learning by creation of a mid level representation between different classes. This representation can also be constructed for land cover data and used to detect minority classes through shared features. We propose a way for generation of attributes by text mining for one of the biggest land cover datasets. With these attributes we achieve state-of-the-art performance in land cover classification and improve results especially for minority classes. Further, we show that these attributes have great potential in weakly supervised land cover segmentation.

Land cover classification is often only looked at from a classification perspective or either coarse or only local maps are used to teach automated approaches to map orbital images. In this work we complement a large remote sensing archive used for multi-label classification with pixel-synchronous land cover maps. The complementary annotations uncover a significant amount of wrongly labelled samples and yield novel insights into the shortcomings of multi-label based approaches. Further, it is now possible to train deep networks for land cover classification with pixel-wise supervision on a large scale.

Recent advances in automated image analysis have lead to an increased number of proposed datasets in remote sensing applications. This permits the successful employment of data hungry state-of-the-art deep neural networks. However, the Earth is not covered equally by semantically meaningful classes. Thus, many land cover datasets suffer from a severe class imbalance. We show that by taking appropriate measures, the performance in the minority classes can be improved by up to 20 percent without affecting the performance in the majority classes strongly. Additionally, we investigate the use of an attribute encoding scheme to represent the inherent class hierarchies commonly observed in land cover analysis.