Food security is at stake, with climate change heavily impacting agriculture and food production. In the present context of extreme events and changing conditions, developing advanced crop yield models can learn from all available information, and providing interpretable predictions for decision-making is of paramount relevance. This work explores the potential and limitations of developing interpretable crop yield models using long short-term memory (LSTM) neural networks, which typically excel at extracting information from time series. LSTMs were designed and trained with multisource satellite and meteorological time series over Continental US (CONUS) and corn, soybean, and wheat yield data from the US Department of Agriculture. Two recent attribution methods are used to interpret and extract knowledge from the developed models: integrated gradients (IG), based on back-propagation, and Shapley (SHAP) values, based on perturbations. Our results show that: 1) LSTM models achieved high accuracy ($\text {R}^{2}>0.56$); 2) multisource combinations outperformed single-variable models in all crop models; 3) both attribution methods were equivalent in detecting essential drivers and their contribution; 4) satellite estimates of enhanced vegetation index (EVI) and vegetation optical depth (VOD) together with meteorological estimates of maximum temperature (TMX) were the most relevant input features for crop yield estimations; and finally and 5) we discovered critical periods of the crop growth cycle for the corn, soybean, and wheat models. The suggested strategy constitutes an important step toward modeling and understanding crop production systems and advancing in automatic data-driven and accountable field management.

Hyperspectral satellite images provide highly-resolved spectral information for large areas. However, spaceborne imaging spectrometers are expensive and currently only a few hyperspectral satellites are in operation. This is a strong limitation, since hyperspectral satellite data provide vital information for numerous fields of application. To overcome this, we developed an emulator using machine learning techniques to generate a synthetic hyperspectral satellite image based on the relationship of a Sentinel-2 (S2) scene and a hyperspectral HyPlant airborne image. The proposed approach was tested on data sets recorded from an agricultural region in western Germany and the results show that a reliable hyperspectral image with the spectral resolution of HyPlant and the spatial extent of the S2 scene can be generated. We systematically tested the approach for different spatial resolutions, including and excluding the S2 spectral bands B1 (coastal aerosol band) and B10 (cirrus band), different machine learning regression algorithms and different numbers of training samples. The best performing parameters were: excluding B1 and B10 bands, resample to 20m and train the emulator with a Neural Networks (NN) with 100'000 samples. That emulator was then applied to the L2A (bottom-of-atmosphere reflectance) S2 subset, and obtained hyperspectral reflectance data were then compared to a reference HyPlant reflectance image of the same region. The synthetic hyperspectral S2-like map was generated quickly and a good agreement with the reference reflectance was achieved. To evaluate the result image we selected the band located at 760 nm due to its importance for the retrieval of solar-induced fluorescence. Goodness-of-fit results (R2of 0.92 and NRMSE of 3.87%) suggest that hyperspectral S2-like reflectance scenes can be produced with high accuracy. The emulator was then applied to a full S2 tile to generate a hyperspectral S2-like reflectance scene (60 Gb), which took less than one hour.

Machine learning models alone are excellent approximators, but very often do not respect the most elementary laws of physics, like mass or energy conservation, so consistency and confidence are compromised. In this paper we describe the main challenges ahead in the field, and introduce several ways to live in the Physics and machine learning interplay: encoding differential equations from data, constraining data-driven models with physics-priors and dependence constraints, improving parameterizations, emulating physical models, and blending data-driven and process-based models. This is a collective long-term AI agenda towards developing and applying algorithms capable of discovering knowledge in the Earth system.

Current anomaly detection (AD) algorithms are typically challenged by either accuracy or efficiency. More accurate nonlinear detectors are typically slow and not scalable. In this letter, we propose two families of techniques to improve the efficiency of the standard kernel Reed-Xiaoli (KRX) method for AD by approximating the kernel function with either the data-independent random Fourier features or the data-dependent basis with the Nyström approach. We compare all methods for both real multi- and hyperspectral images. We show that the proposed efficient methods have a lower computational cost, and they perform similar to (or outperform) the standard KRX algorithm thanks to their implicit regularization effect. Last but not least, the Nyström approach has an improved power of detection.

Modeling and understanding the Earth system is a constant and challenging scientific endeavour. When a clear mechanistic model is unavailable, complex or uncertain, learning from data can be an alternative. While machine learning has provided excellent methods for detection and retrieval, understanding the governing equations of the system from observational data seems an elusive problem. In this paper we introduce sparse regression to uncover a set of governing equations in the form of a system of ordinary differential equations (ODEs). The presented method is used to explicitly describe variable relations by identifying the most expressive and simplest ODEs explaining data to model relevant components of the biosphere.

High spatial resolution vegetation products are fundamental in different fields, such as improving the understanding of crop seasonality at regional scales. Here, two new vegetation products such as the Leaf Area Index (LAI) and the Fraction of Absorbed Photosynthetically Active Radiation (FAPAR) are downscaled at continental scales. A novel HIghly Scalable Temporal Adaptive Reflectance Fusion Model (HIS-TARFM) is used to generate the gap-free time series of Landsat surface reflectance data by fusing MODIS and Landsat reflectance for the contiguous United States. An artificial neural network is trained to capture the relationship between the gap free Landsat surface reflectance and the MODIS LAI/FAPAR products and allows to predict both biophysical variables at 30 meters spatial resolution. The results confirm that both vegetation products largely agree with the test dataset, providing low error and high explained variance.

In this work we propose the use of Long Short-Term Memory (LSTM) Recurrent Neural Networks for multivariate time series of satellite data for crop yield estimation. Recurrent nets allow exploiting the temporal dimension efficiently, but interpretability is hampered by the typically overparameterized models. The focus of the study is to understand LSTM models by looking at the hidden units distribution, the impact of increasing network complexity, and the relative importance of the input covariates. We extracted time series of three variables describing the soil-vegetation status in agroe-cosystems -soil moisture, VOD and EVI- from optical and microwave satellites, as well as available in situ surveys on crops across Continental U.S. to perform the experiments. Firstly, the models were validated in error terms. Secondly, the trained models were visualized and, thirdly, some useful statistics were extracted from the hidden unit activation heatmaps, accounting for redundancy and cluttering of activation responses. Results reveal how networks assign most of the relevance to soil moisture and focus on two phenological stages of crop growth.

In this work, we propose to exploit both the temporal and spatial correlations in Earth observation satellite images through deep learning methods. In particular, the combination of a U-Net convolutional neural network together with a convolutional long short-term memory (LSTM) layer is proposed. This model is applied for cloud detection on MSG/SEVIRI image time series over selected landmarks. Implementation details are provided and our proposal is compared against a standard SVM and a U-Net without the convolutional LSTM layer but including temporal information too. Experimental results show that this combination of networks exploits both the spatial and temporal dependence and provides state-of-the-art classification results on this dataset.

In many remote sensing applications, one wants to estimate variables or parameters of interest from observations. When the target variable is available at a resolution that matches the remote sensing observations, standard algorithms, such as neural networks, random forests, or the Gaussian processes, are readily available to relate the two. However, we often encounter situations where the target variable is only available at the group level, i.e., collectively associated with a number of remotely sensed observations. This problem setting is known in statistics and machine learning as multiple instance learning (MIL) or distribution regression (DR). This article introduces a nonlinear (kernel-based) method for DR that solves the previous problems without making any assumption on the statistics of the grouped data. The presented formulation considers distribution embeddings in reproducing kernel Hilbert spaces and performs standard least squares regression with the empirical means therein. A flexible version to deal with multisource data of different dimensionality and sample sizes is also presented and evaluated. It allows working with the native spatial resolution of each sensor, avoiding the need for matchup procedures. Noting the large computational cost of the approach, we introduce an efficient version via random Fourier features to cope with millions of points and groups. Real experiments involve the Soil Moisture Active Passive (SMAP) vegetation optical depth (VOD) data for the estimation of crop production in the U.S. Corn Belt and the Moderate Resolution Imaging Spectroradiometer (MODIS) and Multi-angle Imaging SpectroRadiometer (MISR) reflectances for the estimation of aerosol optical depth (AOD). An exhaustive empirical evaluation of the method is done against naive (linear and nonlinear) approaches based on input-space means as well as previously presented methods for MIL. We provide source code of our methods in http://isp.uv.es/code/dr.html.

Establishing causal relations between random variables from observational data is perhaps the most important challenge in today's science. In remote sensing and geosciences, this is of special relevance to better understand the earth's system and the complex interactions between the governing processes. In this paper, we focus on an observational causal inference, and thus, we try to estimate the correct direction of causation using a finite set of empirical data. In addition, we focus on the more complex bivariate scenario that requires strong assumptions and no conditional independence tests can be used. In particular, we explore the framework of (nondeterministic) additive noise models, which relies on the principle of independence between the cause and the generating mechanism. A practical algorithmic instantiation of such principle only requires: 1) two regression models in the forward and backward directions and 2) the estimation of statistical independence between the obtained residuals and the observations. The direction leading to more independent residuals is decided to be the cause. We instead propose a criterion that uses the sensitivity (derivative) of the dependence estimator, the sensitivity criterion allows to identify samples most affecting the dependence measure, and hence, the criterion is robust to spurious detections. We illustrate the performance in a collection of 28 geoscience causal inference problems, a database of radiative transfer models simulations and machine learning emulators in vegetation parameter modeling involving 182 problems, and assessing the impact of different regression models in a carbon cycle problem. The criterion achieves the state-of-the-art detection rates in all cases, and it is generally robust to noise sources and distortions. The presented approach confirms the validity in observational bivariate problems in the earth sciences.

This paper introduces a novel statistical regression framework that allows the incorporation of consistency constraints. A linear and nonlinear (kernel-based) formulation are introduced, and both imply closed-form analytical solutions. The models exploit all the information from a set of drivers while being maximally independent of a set of auxiliary, protected variables. We successfully illustrate the performance in the estimation of chlorophyll content.

In this work we propose a method to find anomalous changes in remote sensing images based on the chronochrome approach. A regressor between images is used to discover the most influential points in the observed data. Typically, the pixels with largest residuals are decided to be anomalous changes. In order to find the anomalous pixels we consider the Cook distance and propose its nonlinear extension using random Fourier features as an efficient nonlinear measure of impact. Good empirical performance is shown over different multispectral images both visually and quantitatively evaluated with ROC curves.

This work tackles the target detection problem through the well-known global RX method. The RX method models the clutter as a multivariate Gaussian distribution, and has been extended to nonlinear distributions using kernel methods. While the kernel RX can cope with complex clutters, it requires a considerable amount of computational resources as the number of clutter pixels gets larger. Here we propose random Fourier features to approximate the Gaussian kernel in kernel RX and consequently our development keep the accuracy of the nonlinearity while reducing the computational cost which is now controlled by an hyperparameter. Results over both synthetic and real-world image target detection problems show space and time efficiency of the proposed method while providing high detection performance.

We present a deep learning model with temporal memory to detect clouds in image time series acquired by the Seviri imager mounted on the Meteosat Second Generation (MSG) satellite. The model provides pixel-level cloud maps with related confidence and propagates information in time via a recurrent neural network structure. With a single model, we are able to outline clouds along all year and during day and night with high accuracy.

This letter introduces warped Gaussian process (WGP) regression in remote sensing applications. WGP models output observations as a parametric nonlinear transformation of a GP. The parameters of such a prior model are then learned via standard maximum likelihood. We show the good performance of the proposed model for the estimation of oceanic chlorophyll content from multispectral data, vegetation parameters (chlorophyll, leaf area index, and fractional vegetation cover) from hyperspectral data, and in the detection of the causal direction in a collection of 28 bivariate geoscience and remote sensing causal problems. The model consistently performs better than the standard GP and the more advanced heteroscedastic GP model, both in terms of accuracy and more sensible confidence intervals.

Current remote sensing image classification problems have to deal with an unprecedented amount of heterogeneous and complex data sources. Upcoming missions will soon provide large data streams that will make land cover/use classification difficult. Machine-learning classifiers can help at this, and many methods are currently available. A popular kernel classifier is the Gaussian process classifier (GPC), since it approaches the classification problem with a solid probabilistic treatment, thus yielding confidence intervals for the predictions as well as very competitive results to the state-of-the-art neural networks and support vector machines. However, its computational cost is prohibitive for large-scale applications, and constitutes the main obstacle precluding wide adoption. This paper tackles this problem by introducing two novel efficient methodologies for GP classification. We first include the standard random Fourier features approximation into GPC, which largely decreases its computational cost and permits large-scale remote sensing image classification. In addition, we propose a model which avoids randomly sampling a number of Fourier frequencies and alternatively learns the optimal ones within a variational Bayes approach. The performance of the proposed methods is illustrated in complex problems of cloud detection from multispectral imagery and infrared sounding data. Excellent empirical results support the proposal in both computational cost and accuracy.

Passive Millimeter Wave Images (PMMWIs) are being increasingly used to identify and localize objects concealed under clothing. Taking into account the quality of these images and the unknown position, shape, and size of the hidden objects, large data sets are required to build successful classification/detection systems. Kernel methods, in particular Gaussian Processes (GPs), are sound, flexible, and popular techniques to address supervised learning problems. Unfortunately, their computational cost is known to be prohibitive for large scale applications. In this work, we present a novel approach to PMMWI classification based on the use of Gaussian Processes for large data sets. The proposed methodology relies on linear approximations to kernel functions through random Fourier features. Model hyperparameters are learned within a variational Bayes inference scheme. Our proposal is well suited for real-time applications, since its computational cost at training and test times is much lower than the original GP formulation. The proposed approach is tested on a unique, large, and real PMMWI database containing a broad variety of sizes, types, and locations of hidden objects.

This paper presents an efficient methodology for approximating kernel functions in Gaussian process classification (GPC). Two models are introduced. We first include the standard random Fourier features (RFF) approximation into GPC, which largely improves the computational efficiency and permits large scale remote sensing data classification. In addition, we develop a novel approach which avoids randomly sampling a number of Fourier frequencies, and alternatively learns the optimal ones using a variational Bayes approach. The performance of the proposed methods is illustrated in complex problems of cloud detection from multispectral imagery.

Establishing causal relations between random variables from observational data is perhaps the most important challenge in today's Science. In remote sensing and geosciences this is of special relevance to better understand the Earth's system and the complex and elusive interactions between processes. In this paper we explore a framework to derive cause-effect relations from pairs of variables via regression and dependence estimation. We propose to focus on the sensitivity (curvature) of the dependence estimator to account for the asymmetry of the forward and inverse densities of approximation residuals. Results in a large collection of 28 geoscience causal inference problems demonstrate the good capabilities of the method.

Distance metric learning aims at obtaining an appropriate metric that conveniently adapts to a particular recognition problem given a set of training pairs. The idea of maximizing a margin that separates similar and dissimilar objects has been used in different ways in several recent works. This paper considers two different learning schemes aiming at the same goal but posing the learning problem either as a batch or as an online formulation. Extensive experiments and the corresponding discussion try to put forward the advantages and drawbacks of each of the approaches considered.