Precise and reliable crop yield prediction serves as a valuable tool empowering farmers to make informed and sustainable decisions. However, yield prediction is intricately challenging due to the various factors that play a role in the complex landscape of crop growth. In this paper, we propose an operational yield forecasting approach based on spatiotemporal Machine Learning and a many-to-many network structure. We demonstrate that the simultaneous consideration of spatial and temporal dependencies of crop yield substantially improves the yield prediction performance on field and subfield level across all regions of our large-scale dataset. We further show how our many-to-many network structure leads to outstanding operational results.

Accurate and reliable crop yield prediction is a complex task. The yield of a crop depends on a variety of factors whose accurate measurement and modeling is challenging. At the same time, reliable yield prediction is highly desirable for farmers to optimize crop production. In this paper, we introduce a modeling based on remote sensing data and Machine Learning models evaluated on a large-scale dataset to address the challenge of an operational crop yield estimation and forecasting on field and subfield level. With our approach, we aim towards a global yield modeling based on Machine Learning models which operates across crop types without the need for crop-specific modeling. We demonstrate that our approach learns to map in-field variability for all studied crop types. Overall, the predictions have an error (RRMSE) of around 15% and an R2value of 0.77 at field level.

Numerous remote sensing applications rely on temporal satellite data, and Deep learning models are increasingly being used for such tasks. Nevertheless, these models operate as black boxes, lacking transparency and understandability. We address this gap by using explainable AI on an agricultural task. Specifically, we trained a recurrent neural network on individual pixels from multispectral time-series of Sentinel-2 satellite images to predict crop yield. We then applied nine feature attribution methods on a sample of the dataset and computed the spectral and temporal contributions to the final individual predictions. The aggregated results were evaluated qualitatively and quantitatively. Results suggest that LIME and Shapley sampling value methods performed best on the quantitative scores, followed by GradientShap. Most backpropagation-based techniques had highly inconsistent scores across the explained data points. Finally, to guide remote sensing practitioners in using Explainable AI on similar datasets, we further discuss some selection criteria to be considered.

We introduce a simple yet effective early fusion method for crop yield prediction that handles multiple input modalities with different temporal and spatial resolutions. We use high-resolution crop yield maps as ground truth data to train crop and machine learning model agnostic methods at the sub-field level. We use Sentinel-2 satellite imagery as the primary modality for input data with other complementary modalities, including weather, soil, and DEM data. The proposed method uses input modalities available with global coverage, making the framework globally scalable. We explicitly highlight the importance of input modalities for crop yield prediction and emphasize that the best-performing combination of input modalities depends on region, crop, and chosen model.

Modern combine harvesters can collect geo-located real-time yield measurement while harvesting. This data can be used to train Machine Learning models that predict the yield at sub-field level based on remote sensing input data. The performance of these models is, however, highly dependent on the quality of the yield data. It is therefore important to develop automatic cleaning techniques to correct for common errors in combine harvester yield maps. In this work, we compare different combinations of data cleaning techniques by evaluating their impact on the yield-prediction model performance at field and sub-field level. Our findings indicate that basic cleaning techniques such as absolute thresholds are sufficient at the field level, whereas the performance at the sub-field level is enhanced through the utilization of more intricate statistical cleaning methods.

In the remote sensing community, Land Use Land Cover (LULC) classification with satellite imagery is a main focus of current research activities. Accurate and appropriate LULC classification, however, continues to be a challenging task. In this paper, we evaluate the performance of multi-temporal (monthly time series) compared to mono-temporal (single time step) satellite images for multi-label classification using supervised learning on the RapidAI4EO dataset. As a first step, we trained our CNN model on images at a single time step for multi-label classification, i.e. mono-temporal. We incorporated time-series images using a LSTM model to assess whether or not multi-temporal signals from satellites improves CLC classification. The results demonstrate an improvement of approximately 0.89% in classifying satellite imagery on 15 classes using a multi-temporal approach on monthly time series images compared to the mono-temporal approach. Using features from multi-temporal or mono-temporal images, this work is a step towards an efficient change detection and land monitoring approach.

Under the sponsorship of the European Union's Horizon 2020 program, RapidAI4EO will establish the foundations for the next generation of Copernicus Land Monitoring Service (CLMS) products. The project aims to provide intensified monitoring of Land Use (LU), Land Cover (LC), and LU change at a much higher level of detail and temporal cadence than it is possible today. Focus is on disentangling phenology from structural change and in providing critical training data to drive advancement in the Copernicus community and ecosystem well beyond the lifetime of this project. To this end we are creating the densest spatiotemporal training sets ever by fusing open satellite data with Planet imagery at as many as 500,000 patch locations over Europe and delivering high resolution daily time series at all locations. We plan to open source these datasets for the benefit of the entire remote sensing community.

Informal settlements are home to the most socially and economically vulnerable people on the planet. In order to deliver effective economic and social aid, non-government organizations (NGOs), such as the United Nations Children's Fund (UNICEF), require detailed maps of the locations of informal settlements. However, data regarding informal and formal settlements is primarily unavailable and if available is often incomplete. This is due, in part, to the cost and complexity of gathering data on a large scale. To address these challenges, we, in this work, provide three contributions. 1) A brand new machine learning dataset purposely developed for informal settlement detection. 2) We show that it is possible to detect informal settlements using freely available low-resolution (LR) data, in contrast to previous studies that use very-high resolution~(VHR) satellite and aerial imagery, something that is cost-prohibitive for NGOs. 3) We demonstrate two effective classification schemes on our curated data set, one that is cost-efficient for NGOs and another that is cost-prohibitive for NGOs, but has additional utility. We integrate these schemes into a semi-automated pipeline that converts either a LR or VHR satellite image into a binary map that encodes the locations of informal settlements.

The increased availability of high-resolution satellite imagery allows to sense very detailed structures on the surface of our planet. Access to such information opens up new directions in the analysis of remote sensing imagery. While deep neural networks have achieved significant advances in semantic segmentation of high-resolution images, most of the existing approaches tend to produce predictions with poor boundaries. In this paper, we address the problem of preserving semantic segmentation boundaries in high-resolution satellite imagery by introducing a novel multi-task loss. The loss leverages multiple output representations of the segmentation mask and biases the network to focus more on pixels near boundaries. We evaluate our approach on the large-scale Inria Aerial Image Labeling Dataset which contains high-resolution images. Our results show that we are able to outperform state-of-the-art methods by 9.8% on the Intersection over Union (IoU) metric without any additional post-processing steps. Source code and all models will be available under https://github.com/bbischke/MultiTaskBuildingSegmentation.

In this paper, we analyze the feasability of learning a latent embedding space from aerial and satellite imagery in order to capture semantic properties of geographical locations. We show that deep neural network, trained with a triplet loss function, can be effectively used to obtain a location-specific embedding. Considering the problem of building footprint segmentation from aerial imagery of varying cities, we leverage these embeddings together with a clustering for the training of location-specific segmentation networks and the selection of the corresponding segmentation network during inference time. We evaluate our approach on the large-scale Inria Aerial Image Labeling Dataset which contains aerial images of globally distributed cities. Our approach achieves an outperformance against state-of-the-art approaches on the Intersection over Union metric for the building class over all cities and by more than 2% for specific cities.

In this paper, we describe a multi-scale machine learning approach to estimate socio-economic attributes of citizens based on the analysis of aerial images. To analyse the effectiveness of the proposed approach we predict building property value classes. The classification of these building property values is a proxy for the socio-economic status of the residents. The approach is based on the fusion of deep Convolutional Neural Networks (CNNs). We compare the proposed approach with non-image and single-scale CNN approaches and demonstrate the effectiveness in a case study using statistical data collected in the city of Amsterdam, Netherlands. We show that the proposed multi-scale approach outperforms the baseline methods.

In this paper, we present how multi-spectral Sentinel-2 satellite images can be used in a machine learning approach based on an encoder-decoder semantic segmentation network to map human settlements. We show the effectiveness of the proposed CNN approach for the mapping of settlements in experiments with 785 European cities. The proposed approach to learn a settlement mapping with noisy ground truth data results in an effective settlement segmentation network with a mean intersection over union of 80.55% and a pixel accuracy of 87.40%.

The integration of information acquired with different modalities, spatial resolution and spectral bands has shown to improve predictive accuracies. Data fusion is therefore one of the key challenges in remote sensing. Most prior work focusing on multi-modal fusion, assumes that modalities are always available during inference. This assumption limits the applications of multi-modal models since in practice the data collection process is likely to generate data with missing, incomplete or corrupted modalities. In this paper, we show that Generative Adversarial Networks can be effectively used to overcome the problems that arise when modalities are missing or incomplete. Focusing on semantic segmentation of building footprints with missing modalities, our approach achieves an improvement of about 2% on the Intersection over Union (IoU) against the same network that relies only on the available modality.

We propose a novel loss function for the training of deep Convolutional Neural Networks (CNNs) focusing on land use and land cover classification in remote sensed data. In satellite imagery, object classes are often highly imbalanced leading to poor pixel-wise classification results when using standard training methods only. In this work, we introduce a loss function which leverages the per class uncertainty of the model during training together with median frequency balancing of the class pixels. We evaluate our result on aerial images of the state-of-the-art dataset Vaihingen. We obtain a significant improvement of the F1-Score and pixel accuracy against the standard cross entropy loss on the small car class. The overall Fl-Score using a single CNN achieves 89.35% resulting in an error reduction of 21.22% against the baseline.

In this paper, we address the challenge of land use and land cover classification using Sentinel-2 satellite images. The key contributions are as follows. We present a novel dataset based on Sentinel-2 satellite images covering 13 different spectral bands and consisting of 10 classes with in total 27,000 labeled images. We evaluate state-of-the-art deep Convolutional Neural Networks (CNNs) on this novel dataset with its different spectral bands. We also evaluate deep CNNs on existing remote sensing datasets and compare the obtained results. With the proposed novel dataset, we achieved an overall classification accuracy of 98.57%. The classification system resulting from the proposed research opens a gate towards various Earth observation applications. We demonstrate how the classification system can assist in improving geographical maps.