Abstract. In this paper, we propose a new data assimilation approach based on iterative filtering and smoothing in the expectation-maximization fashion, incorporating optical flow prediction models for dynamic state estimation. The concept is suitable for applications where state estimation relies on two-dimensional images and where no natural physical prediction model is available. We apply the proposed approach to dynamic X-ray images and both real and synthetic satellite data, demonstrating that iterative filtering and smoothing with optical flow models yields improved results compared to using an identity model approach. The quality of the state estimates improves both visually and in terms of the root mean squared error, typically after a couple of iterations. However, in some cases, continued iterations may lead to deteriorating results. Therefore, monitoring the quality of both optical flow and state estimates is crucial in the iterative approach.

Multiplane Images (MPIs) have shown to be excellent scene representations to synthesize new scene views. Indeed, MPIs are able to model challenging occlusions and reflections, and allow to render novel images in real time and with angular consistency. However, their memory footprint constitutes their major limitation. In this work, we propose a learning-based method that computes compact and adaptive MPIs. Our network promotes sparsity in the MPIs to only keep the necessary scene information. Besides, we adapt the depth sampling to the given scene to optimize the available memory and increase the synthesis quality with a restricted number of planes. Moreover, in contrast to recent work, our approach does not need individual training per scene and is able to generalize well to unseen scenarios. An extensive evaluation shows the superiority of our approach with respect to the state of the art on diverse view synthesis datasets.

Atmospheric radiative transfer models (RTMs) simulate the light propagation in the Earth's atmosphere. With the evolution of RTMs, their increase in complexity makes them impractical in routine processing such as atmospheric correction. To overcome their computational burden, standard practice is to interpolate a multidimensional lookup table (LUT) of prestored simulations. However, accurate interpolation relies on large LUTs, which still implies large computation times for their generation and interpolation. In recent years, emulation has been proposed as an alternative to LUT interpolation. Emulation approximates the RTM outputs by a statistical regression model trained with a low number of RTM runs. However, a concern is whether the emulator reaches sufficient accuracy for atmospheric correction. Therefore, we have performed a systematic assessment of key aspects that impact the precision of emulating MODTRAN: 1) regression algorithm; 2) training database size; 3) dimensionality reduction (DR) method and a number of components; and 4) spectral resolution. The Gaussian processes regression (GPR) was found the most accurate emulator. The principal component analysis remains a robust DR method and nearly 20 components reach sufficient precision. Based on a database of 1000 samples covering a broad range of atmospheric conditions, GPR emulators can reconstruct the simulated spectral data with relative errors below 1% for the 95th percentile. These emulators reduce the processing time from days to minutes, preserving sufficient accuracy for atmospheric correction and providing model uncertainties and derivatives. We provide a set of guidelines and tools to design and generate accurate emulators for satellite data processing applications.

Recently, learning methods have been designed to create Multiplane Images (MPIs) for view synthesis. While MPIs are extremely powerful and facilitate high quality renderings, a great amount of memory is required, making them impractical for many applications. In this paper, we propose a learning method that optimizes the available memory to render compact and adaptive MPIs. Our MPIs avoid redundant information and take into account the scene geometry to determine the depth sampling.

Because light field devices have a limited angular resolution, artificially reconstructing intermediate views is an interesting task. In this work, we propose a novel way to solve this problem using deep learning. In particular, the use of Long Short Term Memory Networks on a plane sweep volume is proposed. The approach has the advantage of having very few parameters and can be run on sequences with arbitrary length. We show that our approach yields results that are competitive with the state-of-the-art for dense light fields. Experimental results also show promising results with light fields with wider baselines.

Physically based radiative transfer models (RTMs) are widely used in Earth observation to understand the radiation processes occurring on the Earth's surface and their interactions with water, vegetation, and atmosphere. Through continuous improvements, RTMs have increased in accuracy and representativity of complex scenes at expenses of an increase in complexity and computation time, making them impractical in various remote sensing applications. To overcome this limitation, the common practice is to precompute large lookup tables (LUTs) for their later interpolation. To further reduce the RTM computation burden and the error in LUT interpolation, we have developed a method to automatically select the minimum and optimal set of input-output points (nodes) to be included in an LUT. We present the gradient-based automatic LUT generator algorithm (GALGA), which relies on the notion of an acquisition function that incorporates: 1) the Jacobian evaluation of an RTM and 2) the information about the multivariate distribution of the current nodes. We illustrate the capabilities of GALGA in the automatic construction and optimization of MODTRAN-based LUTs of different dimensions of the input variables space. Our results indicate that when compared with a pseudorandom homogeneous distribution of the LUT nodes, GALGA reduces:1) the LUT size by >24%; 2) the computation time by 27%; and 3) the maximum interpolation relative errors by at least 10%. It is concluded that an automatic LUT design might benefit from the methodology proposed in GALGA to reduce interpolation errors and computation time in computationally expensive RTMs.

Accurately disentangling the tiny Solar–Induced Chlorophyll Fluorescence (SIF) from canopy reflected solar irradiance by using passive remote sensing techniques is always challenging. Regardless the scale at which SIF is measured, i.e., proximal sensing, airborne or satellite level; instrumental and atmospheric effects must be accounted for and compensated as part of the SIF retrieval strategy. Regarding the instrumental effects, the use of very high spectral resolution spectrometers makes mandatory an accurate characterization of the Instrument Spectral Response Function (ISRF); and – in the case of imager spectrometers – an accurate characterization of the full instrument response in the spectral and in the spatial domain. Otherwise, spectral distortions derived by an inaccurate instrument's response characterization would rapidly derive into errors on SIF estimations. With regards to the atmospheric effects, when exploiting the oxygen absorption features to estimate SIF, aerosol characterization generally becomes the bottleneck of the atmospheric correction strategy due to its strong impact on these spectral regions. In this work, instrumental and atmospheric effects are analyzed twofold: (1) focusing on their detection by inspecting the at–sensor radiance and surface apparent reflectance, and (2) proposing an atmospheric and instrument compensation strategy to mitigate these effects.

The contribution presents the development and testing of a fluorescence retrieval scheme based on the ESA's FLuorescence EXplorer mission concept. The algorithm employs on a coupled surface-atmosphere forward model at oxygen absorption bands: i) the atmospheric effect is computed by MODTRAN5; ii) the surface reflectance and fluorescence are modeled by means of the Spectral Fitting approach. The algorithm, previously tested on numerical simulations, was further implemented and optimized to process real observations collected by the FLEX airborne demonstrator HyPlant. The retrieval scheme has been tested on a number of flight lines collected in several locations, different ecosystems types, atmospheric conditions and instrument observation conditions. For the first time, this work shows the capability of retrieving canopy fluorescence from real airborne observations by means of a physically-based algorithm as envisaged for FLEX. The results achieved on the large core data sets of imageries show the consistency of the physical retrieval algorithm for a wide range of scenarios and fluorescence values are in line with ground observations.

An End-to-end simulator (E2ES) is a tool to evaluate the performance of a satellite mission. Once a mission is approved for operation, E2ES evolves during Phase C/D to become a supporting tool for the development and validation of the ground data processor, as well as for simulating data sets to test the Prototype and Operational Processors. FLEX-E is the E2ES of the FLEX/Sentinel-3 tandem mission, which was selected in 2015 as ESA's eighth Earth Explorer. The FLEX-E evolution implies the consolidation of all the retrieval algorithms (e.g. fluorescence, reflectance, biophysical variables), the implementation of new scientific developments, as well the improvement of the co-registration process, the atmospheric correction, and the retrieval of the Level-2 products. The high-level modular design of the improved Level-2 retrieval module will permit a detailed analysis of the errors propagating through the processing chain.

ESA's FLEX/Sentinel-3 tandem mission aims at mapping Sun-induced fluorescence (SIF) as a proxy to quantify photosynthetic activity of terrestrial vegetation. Due to the complexity of the mission concept and stringent requirements for the data processing algorithms, ESA developed a Phase A/B1 End-to-End Mission Performance Simulator (E2ES) tool to reproduce the expected mission performance and check the mission and instrument concepts. In the current Phase C/D, the E2ES concept must evolve to consolidate the whole data processing chain, providing an accurate figures of the whole mission error budget and serving as a roadmap for the future development of FLEX Ground Segment. This paper gives an overview of the activities involving the FLEX E2ES, from the conceptual development phase (Phase A/B1) to the implementation of the Ground Segment and operations (Phase C/D). Particularly, we will focus on the high-level description of the Level-2 data processing chain from an architecture and software design point of view.

End-to-end mission performance simulators (E2ES) are suitable tools to accelerate satellite mission development from concet to deployment. One core element of these E2ES is the generation of synthetic scenes that are observed by the various instruments of an Earth Observation mission. The generation of these scenes rely on Radiative Transfer Models (RTM) for the simulation of light interaction with the Earth surface and atmosphere. However, the execution of advanced RTMs is impractical due to their large computation burden. Classical interpolation and statistical emulation methods of pre-computed Look-Up Tables (LUT) are therefore common practice to generate synthetic scenes in a reasonable time. This work evaluates the accuracy and computation cost of interpolation and emulation methods to sample the input LUT variable space. The results on MONDTRAN-based top-of-atmosphere radiance data show that Gaussian Process emulators produced more accurate output spectra than linear interpolation at a fraction of its time. It is concluded that emulation can function as a fast and more accurate alternative to interpolation for LUT parameter space sampling.

During the design phase of a satellite mission, end-to-end mission performance simulator (E2ES) tools allow scientists and engineers evaluating the mission concept, consolidating system technical requirements and analyzing the suitability of the implemented technical solutions and data processing algorithms. The generation of synthetic scenes is one of the core parts of an E2ES, providing scenes (ground truth) as would be observed by satellite instruments and used as reference against simulated retrieved mission products. An appropriate generation of the scene also allows assessing the performance of the ground data processing chain replacing real instrument data before the mission is in orbit, for which the fidelity of the scene generation is critical. This paper describes the design of a generic scene generator (GSG) with capabilities to generate complex 3-D synthetic scenes that combine the effects of surface, heterogeneity, topography and atmosphere, and viewing/illumination geometry. The proposed design allows generating consistent high spatial and spectral resolution top-of-atmosphere radiance scenes for multiple instruments based on the use of thematic maps, radiative transfer models, and reflectance databases. The described GSG was implemented within the FLuorescence EXplorer (FLEX) E2ES software tool and showed its capabilities to generate compatible scenes for the fluorescence imaging spectrometer, ocean and land color instrument, and sea and land surface temperature radiometer instruments of ESA’s FLEX/Sentinel-3 tandem mission and to validate the fulfillment of the FLEX mission requirements.

Since oxygen (O2) absorption of light becomes more pronounced at higher pressure levels, even a few meters distance between the target and the sensor can strongly affect canopy-leaving Solar-Induced chlorophyll Fluorescence (SIF) retrievals. This study was conducted to quantify the consequent error propagation and the impact of ignoring oxygen absorption effects on proximal sensing SIF measurements based on the O2-A absorption band with field-acquired and simulated data. It was demonstrated that the uncorrected oxygen transmittance between target and sensor distance of 10 m can lead to SIF relative errors ranging from 66% to higher than 100% when using a Spectral Fitting (SF) technique or the 3FLD retrieval method, respectively. A proposed strategy to include oxygen transmittance effects on the well-known 3FLD and SF techniques is presented here and applied to the NASA-GSFC multi-angular spectral system known as FUSION over a field of corn plants (Zea mays L.) during the second half of the 2014 growing season. Daily averages of oxygen-corrected SIF measurements from FUSION were related to daily averages of heat and energy fluxes obtained from a nearby Eddy-Covariance (EC) flux tower, and showed a consistent behaviour with similar experiments performed at leaf level.

This paper presents a novel unsupervised method to transfer the style of an example image to a source image. The complex notion of image style is here considered as a local texture transfer, eventually coupled with a global color transfer. For the local texture transfer, we propose a new method based on an adaptive patch partition that captures the style of the example image and preserves the structure of the source image. More precisely, this example-based partition predicts how well a source patch matches an example patch. Results on various images show that our method outperforms the most recent techniques.

The FLuorescence EXplorer (FLEX) mission, selected as the European Space Agency's eighth Earth Explorer, aims to globally measure the sun-induced-chlorophyll-fluorescence spectral emission from terrestrial vegetation. In the frame of the FLEX mission, several industrial and scientific studies have analyzed the instrument design, image processing algorithms, or modeling aspects. At the same time, a common tool is needed to address the overall FLEX mission performance by combining all these features. For this reason, an end-to-end mission performance simulator has been developed for the FLEX mission (FLEX-E). This paper describes the FLEX-E software design, which combines the generation of complex synthetic scenes with an advanced modeling of the instrument behavior and the full processing scheme up to the final fluorescence product. The results derived from FLEX-E simulations indicate that the instrument and developed image processing algorithms are able to retrieve the sun-induced fluorescence with an accuracy below the 0.2$\text{mW}\cdot\text{m}^{-2}\cdot\text{sr}^{-1}\cdot \text{nm}^{-1}$mission requirement. It is expected that FLEX-E will not only optimize the FLEX retrieval algorithms and technical requirements, but also serve as the baseline for the ground processing implementation and testing of calibration/validation procedures.

This paper addresses the problem of tonal fluctuation in videos. Due to the automatic settings of consumer cameras, the colors of objects in image sequences might change over time. We propose here a fast and computationally light method to stabilize this tonal appearance, while remaining robust to motion and occlusions. To do so, a minimally viable color correction model is used, in conjunction with an effective estimation of dominant motion. The final solution is a temporally weighted correction, explicitly driven by the motion magnitude, both visually efficient and very fast, with potential to real time processing. Experimental results obtained on a variety of sequences outperform the current state of the art in terms of tonal stability, at a much reduced computational complexity.

In this work, we present a fast and parametric method to achieve tonal stabilization in videos containing color fluctuations. Our main contribution is to compensate tonal instabilities with a color transformation guided by dominant motion estimated between temporally distant frames. Furthermore, we propose a temporal weighting scheme, where the intensity of tonal stabilization is directly guided by the motion speed. Experiments show that the proposed method compares favorably with the state-of-the-art in terms of accuracy and computational complexity.

A new fluorescence retrieval method is proposed to support ESA's 8th Earth Explorer FLuorescence EXplorer/Sentinel-3 (FLEX-S3) candidate tandem mission. FLEX is the first mission specially dedicated to measure the Sun-Induced vegetation chlorophyll fluorescence (SIF) strongly related with the vegetation photosynthetic activity. Most hyperspectral fluorescence retrieval algorithms available in the literature are very sensitive to true reflectance modelization and/or they assume the atmospheric status as known. The proposed algorithm delivers the retrieval of full fluorescence spectrum at canopy level by using only Top Of Atmosphere (TOA) radiances from S3 and FLEX as input. Once the spatial co-registration and cross-calibration of S3 and FLEX images have been performed, the proposed method starts with (1) the atmospheric correction of TOA radiances, characterizing the state of the atmosphere, (2) performing a first estimation of fluorescence values in main oxygen absorption bands without any approximation of true reflectance spectrum, and using this fluorescence estimation to initialize a Spectral Fitting Method (SFM) to finally retrieving a full fluorescence spectrum. This proposed fluorescence retrieval method is currently being implemented at the Level-2 Retrieval Module (L2RM) of the FLEX/End-To-End Simulator (E2ES).

The Hyperspectral Imager for the Coastal Ocean (HICO) is an imaging spectrometer designed with a very high signal-to-noise ratio to monitor coastal ocean and inland waters. The processing of Top-Of-Atmosphere radiance data down to surface reflectance is fundamental for the retrieval of water quality products. However, the current HICO processing chain does not provide atmospheric corrected data nor higher-level water quality products. This paper describes the algorithms implemented within an HICO data processing chain that includes image pre-processing, atmospheric correction and the retrieval of water quality parameters. The implemented algorithms have been validated over a set of HICO images showing a good match with in-situ surface reflectance data and correlation (R2= 0.95) between in-situ measured and retrieved Chl-a over a water body. It is expected that the presented algorithms will ease the processing of HICO data down to surface reflectance allowing to derive water quality parameters.

Light-field imaging has been recently introduced to mass market by the hand held plenoptic camera Lytro. Thanks to a microlens array placed between the main lens and the sensor, the captured data contains different views of the scene from different view points. This offers several post-capture applications, e.g., computationally changing the main lens focus. The raw data conversion in such cameras is however barely studied in the literature. The goal of this paper is to study the particularly overlooked problem of demosaicking the views for plenoptic cameras such as Lytro. We exploit the redundant sampling of scene content in the views, and show that disparities estimated from the mosaicked data can guide the demosaicking, resulting in minimum artifacts compared to the state of art methods. Besides, by properly addressing the view demultiplexing step, we take the first step towards light field super-resolution with negligible computational overload.

This paper presents the Contrast Invariant and Affine Optical Flow (CIAO), a dense area-based sub-pixel image matching algorithm. CIAO does not force the estimated disparity field between two images to be smooth and allows for contrast and brightness changes. It is robust to drastic changes in the images' content thanks to an adaptive weighting of the neighboring pixels. In addition, the proposed model considers local affine displacements instead of simpler translations. CIAO proves particularly useful to extract high quality, high accuracy, and high density disparity maps from pairs of stereoscopic images. Comparative results with real and synthetic data are provided.

This paper introduces a statistical method to decide whether two blocks in a pair of images match reliably. The method ensures that the selected block matches are unlikely to have occurred "just by chance." The new approach is based on the definition of a simple but faithful statistical background model for image blocks learned from the image itself. A theorem guarantees that under this model, not more than a fixed number of wrong matches occurs (on average) for the whole image. This fixed number (the number of false alarms) is the only method parameter. Furthermore, the number of false alarms associated with each match measures its reliability. This a contrario block-matching method, however, cannot rule out false matches due to the presence of periodic objects in the images. But it is successfully complemented by a parameterless self-similarity threshold. Experimental evidence shows that the proposed method also detects occlusions and incoherent motions due to vehicles and pedestrians in nonsimultaneous stereo.

This article explores the subpixel accuracy attainable for the disparity computed from a rectified stereo pair of images with small baseline. In this framework we consider translations as the local deformation model between patches in the images. A mathematical study first shows how discrete block-matching can be performed with arbitrary precision under Shannon–Whittaker conditions. This study leads to the specification of a block-matching algorithm which is able to refine disparities with subpixel accuracy. Moreover, a formula for the variance of the disparity error caused by the noise is introduced and proved. Several simulated and real experiments show a decent agreement between this theoretical error variance and the observed root mean squared error in stereo pairs with good signal-to-noise ratio and low baseline. A practical consequence is that under realistic sampling and noise conditions in optical imaging, the disparity map in stereo-rectified images can be computed for the majority of pixels (but only for those pixels with meaningful matches) with a $1/20$ pixel precision.

This paper proposes a statistical rejection rule, designed for small baseline stereo satellites. The method learns an a contrario model for image blocks and discards the casual matches between the images of the stereo pair. A formula estimating the expected number of false alarms under the background model is proved. Comparative experiments on quasi-simultaneous stereo in aerial imagery demonstrate the elimination of all incoherent motions.

The obtention of 3D information from two images requires the perfect control of a long chain of algorithms: internal and external calibration, stereo-rectification, correlation, and finally 3D reconstruction. In this paper we focus on the improvement of the correlation step for small baseline stereo. In that setting a very strong sub-pixel accuracy is possible. This accuracy is also necessary to obtain high resolution urban maps in geographic information systems. We show that if the images are carefully taken, then the disparity map in stereo-rectified images can be computed for a majority of image points to a 1/20 pixel precision under realistic noise conditions. Experiments on the Middlebury benchmark also stress the need for a methodology to create reliable ground truths.