Image-based localization in GNSS-denied environments is critical for UAV autonomy. Existing state-of-the-art approaches rely on matching UAV images to geo-referenced satellite images; however, they typically require large-scale, paired UAV–satellite datasets for training. Such data are costly to acquire and often unavailable, limiting their applicability. To address this challenge, we adopt a training paradigm that removes the need for UAV imagery during training by learning directly from satellite-view reference images. This is achieved through a dedicated augmentation strategy that simulates the visual domain shift between satellite and real-world UAV views. We introduce CAEVL, an efficient model designed to exploit this paradigm, and validate it on ViLD, a new and challenging dataset of real-world UAV images that we release to the community. Our method achieves competitive performance compared to approaches trained with paired data, demonstrating its effectiveness and strong generalization capabilities.

Classical approaches to the rectification of a single image of a product, without stereo correspondences, require spatial landmarks. These landmarks, constructed from high-contrast elementary shapes that can be detected with simple algorithms, are highly conspicuous. To rectify complex deformations, one can use chessboard patterns of markers with elements that break quadrilateral symmetry, such as the three eyes of a QR code. However, these marker boards are even more conspicuous than a single marker. In traceability applications, only one site of marking is used, limiting the complexity of the surface on which it can be read, and exposing the mark to deidentification attacks for diversion of the product to a grey market. We introduce a method for constructing stealth and robust fiducial markers that can be displayed across a surface, limiting exposure to marker tampering for product deidentification. These markers, which we refer to as self-rectifying textures, can be used to rectify complex deformations by solving an inverse problem rather than relying on pixel correspondences of conspicuous landmarks. These stealth textures place fiducial markers in the autocorrelation of the image. In this way, crops of the deformed texture can be rectified using only these spatially invariant statistical properties. Affine transformations of an image correspond to linear transformations of the autocorrelation, without phase component. Exploiting this fact, self-rectifying textures enable the local estimation of the differential of a planar deformation by identifying landmarks in the autocorrelation image, such as peaks, whose locations in the fronto-parallel view of the texture are known. The translation component can be recovered independently via phase correlation. A rectifying map, modulo translations, can also be fit directly to local observations of the differential of the deformation, without access to the rectified texture or need for phase correlation. Self-rectifying textures can be used for communication, watermarking, authentication, surface identification, calibration, and geometry processing.

Even though data annotation is extremely important for interpretability, research, and development of artificial intelligence solutions, annotating data remains costly. Research efforts such as active learning or few-shot learning alleviate the cost by increasing sample efficiency, yet the problem of annotating data more quickly has received comparatively little attention. Leveraging a predictor has been shown to reduce annotation cost in practice but has not been theoretically considered. We ask the following question: to annotate a binary classification dataset with N samples, can the annotator answer less than N yes/no questions? Framing this question-and-answer (Q&A) game as an optimal encoding problem, we find a positive answer given by the Huffman encoding of the possible labelings. Unfortunately, the algorithm is computationally intractable even for small dataset sizes. As a practical method, we propose to minimize a cost function a few steps ahead, similarly to lookahead minimization in optimal control. This solution is analyzed, compared with the optimal one, and evaluated using several synthetic and real-world datasets. The method allows a significant improvement (23-86%) in the annotation efficiency of real-world datasets.

Recently, Gaussian splatting has emerged as a strong alternative to NeRF, demonstrating impressive 3D modeling capabilities while requiring only a fraction of the training and rendering time. In this paper, we show how the standard Gaussian splatting framework can be adapted for remote sensing, retaining its high efficiency. This enables us to achieve state-of-the-art performance in just a few minutes, compared to the day-long optimization required by the best-performing NeRF-based Earth observation methods. The proposed framework incorporates remote-sensing improvements from EO-NeRF, such as radiometric correction and shadow modeling, while introducing novel components, including sparsity, view consistency, and opacity regularizations.

Point cloud visibility estimation is fundamental as it is useful for many computer vision applications including surface reconstruction, 3D segmentation from paired images and point densification. Previous works showed outstanding results on simple object and outdoor datasets. However, unlike the previously studied scenes, the most challenging environments are those providing a high amount of object points in the same direction, typically in complex indoor scenes. In this kind of environments, due to the lack of real data ground truth, quantitative analysis are either missing or based on simulated data. In this work, we present IRIS-VIS (Industrial Room In Saclay - VISibility), a new dataset for point visibility estimation in an indoor environment. It is a high complexity scene due to the large variety in the shape, size and orientation of the objects. To our know-ledge, this is the first dataset on real indoor data providing a dense LiDAR station-based point cloud along with a well-fitted CAD model. The latter is useful to compute automatically, quickly and accurately the visibility from any given viewpoint, enabling evaluations under infinite conditions. We propose new metrics for the visibility estimation task and evaluate state-of-the-art methods in both sparse and dense conditions with the proposed dataset.

This paper introduces a new method for authenticating engravings, crucial to fight counterfeiting and related issues in industries like luxury brands. The approach is based on extracting from the engravings natural minutiae, inspired by human biometrics, called (n, n)-modules. These modules, representing small sub-matrices of the printed pattern, that vary in appearance due to factors like material characteristics and engraving procedure. To create a reliable fingerprint, we proposed a voting system, with each vote being a comparison between two (n, n)-modules. This method leverages of the fact that the defects of an engraving or printing are difficult to reproduce and are observable on the scale of a printed or engraved point. We evaluate two ways of comparing the minutiae: a standard one using Euclidean distance and a deep learning-based one employing a convolutional neural network. The process delivers high accuracy and recall in authenticating engravings and shows robustness under various lighting conditions and levels of image blur.

Satellite imaging has proven to be crucial to monitor methane emissions and help reduce them. In this paper, we propose an automatic practical methodology to use time series from geostationary satellites like GOES-16. While these satellites offer a poor spatial and spectral resolution, their revisit time is unmatched: GOES-16 delivers an image every five minutes for the CONUS region. The proposed approach takes advantage of this fast revisit time to monitor the evolution of large methane emissions. We show the performance on an emission in Mexico and another in the US. This is the first step toward a real time monitoring of methane emissions using geostationary satellite imagery.

Reducing methane emissions is essential to tackle climate change. Here, we address the problem of detecting automatically point source methane leaks using high resolution hyperspectral images from the PRISMA satellite. We propose an improvement of the classical matched filter method by using an adjustment coefficient. We introduce this new method under the name: Model Adjusted Matched Filter (MAMF). We show that the MAMF method reduces the fraction of false detections compared to the Matched Filter (MF) and the Adaptive Cosine Estimator (ACE) without preventing the detection of plumes. To validate the method, we use a dataset of manually annotated plumes on PRISMA images. We then show that our method outperforms the matched filter and the adaptive cosine estimator in terms of F1 score.

We propose a new method for the geolocalization of Un-maned Aerial Vehicles (UAV) in environments without Global Navigation Stallite Systems (GNSS). Current state-of-the-art methods use an offline-trained encoder to compute a vector representation (embedding) of the current UAV’s view, and compare it with the pre-computed embeddings of geo-referenced images in order to deduce the UAV’s position. Here, we show that the performance of these methods can be greatly improved by pre-processing the images by extracting their edges, which are robust to seasonal and illumination changes. Moreover, we also show that using edges improves the robustness to orientation and altitude errors. Finally, we present a confidence criterion for localization. Our findings are validated using synthetic experiments.

The use of NeRF to model 3D scenes from satellite images is becoming increasingly common. However, the models proposed to date assume the availability of pre-processed RGB images as input. This contrasts with the multispectral nature of raw satellite products. Optical satellite sensors do not acquire RGB images but a wider variety of spectral bands, which may have different spatial resolution. We propose a NeRF framework to simultaneously handle panchromatic data and lower resolution spectral bands (e.g., color bands), and investigate the contribution of the low-resolution bands to the output model. Our method achieves comparable or better results with respect to previous approaches that rely on a separate pansharpening step. The model can also be used to generate a pansharpened image surrogate for each input view, as it natively performs super-resolution in the color bands.

Neural radiance fields, or NeRF, represent a breakthrough in the field of novel view synthesis and 3D modeling of complex scenes from multi-view image collections. Numerous recent works have shown the importance of making NeRF models more robust, by means of regularization, in order to train with possibly inconsistent and/or very sparse data. In this work, we explore how differential geometry can provide elegant regularization tools for robustly training NeRF-like models, which are modified so as to represent continuous and infinitely differentiable functions. In particular, we present a generic framework for regularizing different types of NeRFs observations to improve the performance in challenging conditions. We also show how the same formalism can also be used to natively encourage the regularity of surfaces by means of Gaussian or mean curvatures.

Object detection models, a prominent class of machine learning algorithms, aim to identify and precisely locate objects in images or videos. However, this task might yield uneven performances sometimes caused by the objects sizes and the quality of the images and labels used for training. In this paper, we highlight the importance of large objects in learning features that are critical for all sizes. Given these findings, we propose to introduce a weighting term into the training loss. This term is a function of the object area size. We show that giving more weight to large objects leads to improved detection scores across all object sizes and so an overall improvement in Object Detectors performances (+2 p.p. of mAP on small objects, +2 p.p. on medium and +4 p.p. on large on COCO val 2017 with InternImage-T). Additional experiments and ablation studies with different models and on a different dataset further confirm the robustness of our findings.

In this article, we examine the possibility of using the interferometric phase on some fixed corners of storage tanks to infer the tank fill ratio. For the study, we use floating roof tanks for which the fill ratio can be inferred from the floating roof position. We observe a correlation between the phase double difference taken at the fixed roof of neighboring tanks and the fill ratio double difference. We highlight some challenges that require further investigation and the development of adapted InSAR techniques.

In this work, we study the problem of single-image super-resolution (SISR) of Sentinel-2 imagery. We show that thanks to its unique sensor specification, namely the inter-band shift and alias, that deep-learning methods are able to recover fine details. By training a model using a simple L1loss, results are free of hallucinated details. For this study, we build a dataset of pairs of images Sentinel-2/PlanetScope to train and evaluate our super-resolution (SR) model.

Reducing methane emissions is essential to tackle climate change. Here, we address the problem of detecting automatically point source methane leaks using high resolution hyperspectral images from the PRISMA satellite. We use a variation of the Matched Filter (MF) called the Adjusted Spectral Matched Filter (ASMF) to detect methane plumes in satellite images. To remove false positives, the detected plumes are confirmed by comparing their orientation to the wind direction extracted from the standard meteorological reanalysis product ERA5. The ASMF reduces the fraction of false detections compared to the MF and without preventing the detection of plumes. To validate the method, we use a recently proposed dataset of manually annotated plumes on PRISMA images. We also compare our detection rate to the detection rate of methods using deep learning or the standard matched filter. We then show that our method outperforms those methods in terms of F1 score.

The detection of smoke plumes by satellite imagery is a comprehensive research topic that can be used to better monitor activity and emissions from the energy and industrial sectors. In this study, we propose a machine learning methodology based on the extraction of relevant features from Sentinel-2 images to perform industrial smoke plume detection. This computer vision problem is modeled as an image classification task based on the presence or absence of plumes from previously identified sources. A dataset of nearly 17,000 hand-labeled images of smoke plumes for activity classification has been compiled to train and evaluate our detection models. The final Gradient Boosting model only uses the 3 RGB bands of Sentinel-2 and after a post-processing step reaches an accuracy of 95%.

Denoising and demosaicing are two critical components of the image/video processing pipeline. While historically these two tasks have mainly been considered separately, current neural network approaches allow to obtain state-of-the-art results by treating them jointly. However, most existing research focuses in single image or burst joint denoising and demosaicing (JDD). Although related to burst JDD, video JDD deserves its own treatment. In this work we present an empirical exploration of different design aspects of video joint denoising and demosaicing using neural networks. We compare recurrent and non-recurrent approaches and explore aspects such as type of propagated information in recurrent networks, motion compensation, video stabilization, and network architecture. We found that recurrent networks with motion compensation achieve best results. Our work should serve as a strong baseline for future research in video JDD.

Multi-view stereo reconstruction of scenes from satellite images is traditionally performed with a pair-wise stereovision approach: (1) multiple views are grouped into pairs, (2) each pair is processed by two-view stereo methods producing an elevation model or point cloud, lastly (3) the pairwise reconstructions are integrated and filtered to obtain a final result. These steps are organized in a pipeline and the end-to-end performance of reconstructions depends on the behavior of these steps. This work introduces two changes that increase the performance of the reconstructions: a new pair selection approach and a new integration method are presented. The new pair selection replaces commonly used heuristics with a principled criterion that predicts the completeness of a pair based on offline simulations. The presented integration method is based on an iterated bilateral filter. Experiments show that these changes yield a systematic improvement on the performance of the pipeline.

Self-supervised representation learning based on Contrastive Learning (CL) has been the subject of much attention in recent years. This is due to the excellent results obtained on a variety of subsequent tasks (in particular classification), without requiring a large amount of labeled samples. However, most reference CL algorithms (such as SimCLR and MoCo, but also BYOL and Barlow Twins) are not adapted to pixel-level downstream tasks. One existing solution known as PixPro proposes a pixel-level approach that is based on filtering of pairs of positive/negative image crops of the same image using the distance between the crops in the whole image. We argue that this idea can be further enhanced by incorporating semantic information provided by exogenous data as an additional selection filter, which can be used (at training time) to improve the selection of the pixel-level positive/negative samples. In this paper we will focus on the depth information, which can be obtained by using a depth estimation network or measured from available data (stereovision, parallax motion, LiDAR, etc.). Scene depth can provide meaningful cues to distinguish pixels belonging to different objects based on their depth. We show that using this exogenous information in the contrastive loss leads to improved results and that the learned representations better follow the shapes of objects. In addition, we introduce a multi-scale loss that alleviates the issue of finding the training parameters adapted to different object sizes. We demonstrate the effectiveness of our ideas on the Breakout Segmentation on Borehole Images where we achieve an improvement of 1.9% over PixPro and nearly 5% over the supervised baseline. We further validate our technique on the indoor scene segmentation tasks with ScanNet and outdoor scenes with CityScapes (1.6% and 1.1% improvement over PixPro respectively).

Modern Earth observation satellites capture multi-exposure bursts of push-frame images that can be super-resolved via computational means. In this work, we propose a super-resolution method for such multi-exposure sequences, a problem that has received very little attention in the literature. The proposed method can handle the signal-dependent noise in the inputs, process sequences of any length, and be robust to inaccuracies in the exposure times. Furthermore, it can be trained end-to-end with self-supervision, without requiring ground truth high resolution frames, which makes it especially suited to handle real data. Central to our method are three key contributions: i) a base-detail decomposition for handling errors in the exposure times, ii) a noise-level-aware feature encoding for improved fusion of frames with varying signal-to-noise ratio and iii) a permutation invariant fusion strategy by temporal pooling operators. We evaluate the proposed method on synthetic and real data and show that it outperforms by a significant margin existing single-exposure approaches that we adapted to the multi-exposure case.

Since the commissioning phase of Sentinel-1A, several calibration studies have improved the geolocation and geometric modeling of the data. The implementation of the corrections presented in these studies is left to the user. The issues found might be confusing when working with bursts in the interferometric wide swath mode, because the geometric shifts present in the data are not usually the same at the burst boundaries. This might introduce small inconsistencies in a mosaic product if not properly handled, which is especially inconve-nient in high precision applications. This paper proposes a method to account for this effect by resampling the bursts before stitching. The method is validated with experiments on real Sentinel-1 data.

Methane emissions monitoring is essential to control methane pollution. In this paper, we propose an automatic practical methodology using time series to estimate the quantity of methane in a given plume using a multispectral satellite like Sentinel-2. Sentinel-2 proposes a low revisit time, a good spatial resolution and a low acquisition cost. Contrary to previous methods, the proposed approach does not require a manual selection of an optimal reference image. We compared its performance on an oil-and-gas site in Kazakhstan. This is the first step toward an automatic global monitoring system for methane plume detection and quantification with these satellites.

Shape from shading (SfS) enables 3D reconstruction of stockpiles from a single image. However, this method requires proper boundary conditions to work properly. Obtaining such Dirichlet and Neumann conditions is equivalent to a segmentation of the heaps. To get a fast and accurate 3D reconstruction, we propose a simple and interactive segmentation method. SfS is then applied on 0.5-meter resolution Synthetic Aperture Radar (SAR) images with more precise boundary conditions. The results show that prior segmentation is preferable to no segmentation for the stockpiles volume estimation problem. Furthermore, we show that the proposed interactive segmentation method reduces the annotation time needed for such a prior segmentation

Self-supervised training enables the application of deep-learning based methods for multi-image super-resolution of satellite imagery. In this work we propose two improvements on the self-supervised Deep-Shift-and-Add (DSA) method introduced by Nguyen et al. First, we demonstrate how the self-supervised loss of DSA can be extended to provide the image interpreter with a spatially varying parameter to control the trade-off between detail preservation and noise removal at test time. Second, we endow the DSA architecture with a mechanism that enables the network to be robust to outliers produced for example by dead pixels, reflections or registration errors.

Different methods can be applied to satellite images to derive an altitude map from a set of images. In this article we evaluate a set of representative methods from different approaches. We consider true multi-view stereo methods as well as pair-wise ones, classic methods and deep learning based ones, methods already in use on satellite images and others that were originally devised for close range imaging and are adapted to satellite imagery. While deep learning (DL) methods have taken over multi-view stereo reconstruction in the last years, this tendency has not fully reached satellite stereo pipelines that still largely rely on pair-wise classic algorithms. For the comparison, we set-up a framework that allows to interface a DL-based stereo method taken from the computer vision literature with a satellite stereo pipeline. For multi-view stereo algorithms we build on a recently proposed framework originally devised to apply Colmap method to satellite images. Methods are compared on several datasets that include sets of images taken within a few days and sets of images taken months apart. Results show that DL methods have, in general, a good generalization power. In particular, the use of the GANet DL method as the matching step in a pair-wise stereo pipeline is promising as it already performs better than the classic counterpart, even without a specific training.

In this study, we compare the different spectral centering methods (referred to as deramping) for the Sentinel-1 images acquired with the TOPSAR method, in the context of interferometry. The deramping is a necessary step prior to image interpolation. We show the analogy between two different approaches in the literature and propose our own improvements. The proposed deramping method approximately re-centers the spectrum, i.e. a small residual spectral shift remains. We validate our improvements with experiments on Sentinel-1 data, and show that interpolating the images containing this residual spectral shift should not induce considerable errors.

The Rational Polynomial Camera (RPC) model can be used to describe a variety of image acquisition systems in remote sensing, notably optical and Synthetic Aperture Radar (SAR) sensors. RPC functions relate 3D to 2D coordinates and vice versa, regardless of physical sensor specificities, which has made them an essential tool to harness satellite images in a generic way. This article describes a terrain-independent algorithm to accurately derive a RPC model from a set of 3D-2D point correspondences based on a regularized least squares fit. The performance of the method is assessed by varying the point correspondences and the size of the area that they cover. We test the algorithm on SAR and optical data, to derive RPCs from physical sensor models or from other RPC models after composition with corrective functions.

Recent constellations of satellites, including the Skysat constellation, are able to acquire burst of images. This new acquisition mode allows for modern image restoration techniques, including multi-frame super-resolution. As the satellite moves during the acquisition of the burst, elevation changes in the scene translate into noticeable parallax. This parallax hinders the results of the restoration. To cope with this issue, we propose a novel parallax estimation method. The method is composed of a linear$\text{Plane}+\text{Parallax}$decomposition of the apparent motion and a multi-frame optical flow algorithm that exploits all frames simultaneously. Using SkySat L1A images, we show that the estimated per-pixel displacements are important for applying multi-frame super-resolution on scenes containing elevation changes and that can also be used to estimate a coarse 3D surface model.

Image registration is a fundamental tool of remote sensing. The recent proliferatio of earth observation satellites has opened the way to the analysis of long image time series with denser temporal repetition. Given this wealth of images, it is crucial to design automatic tools to process them. We thus propose a method for the global registration of satellite image time series, that leverages their redundancy to improve in precision and robustness. By computing the relative displacement for all possible pairs of images, we are able to discard outliers and minimize the number of misaligned images. Experiments on synthetic data show that longer image series are registered with a higher precision.

This paper proposes a system for automatic surface volume monitoring from time series of SkySat pushframe imagery. A specific challenge of building and comparing large 3D models from SkySat data is to correct inconsistencies between the camera models associated to the multiple views that are necessary to cover the area at a given time, where these camera models are represented as Rational Polynomial Cameras (RPCs). We address the problem by proposing a date-wise RPC refinement, able to handle dynamic areas covered by sets of partially overlapping views. The cameras are refined by means of a rotation that compensates for errors due to inaccurate knowledge of the satellite attitude. The refined RPCs are then used to reconstruct multiple consistent Digital Surface Models (DSMs) from different stereo pairs at each date. RPC refinement strengthens the consistency between the DSMs of each date, which is extremely beneficial to accurately measure volumes in the 3D surface models. The system is tested in a real case scenario, to monitor large coal stockpiles. Our volume estimates are validated with measurements collected on site in the same period of time.

The PROBA-V Super-Resolution challenge distributes real low-resolution image series and corresponding high-resolution targets to advance research on Multi-Image Super Resolution (MISR) for satellite images. However, in the PROBA-V dataset the low-resolution image corresponding to the high-resolution target is not identified. We argue that in doing so, the challenge ranks the proposed methods not only by their MISR performance, but mainly by the heuristics used to guess which image in the series is the most similar to the high-resolution target. We demonstrate this by improving the performance obtained by the two winners of the challenge only by using a different reference image, which we compute following a simple heuristic. Based on this, we propose PROBA-V-REF a variant of the PROBA-V dataset, in which the reference image in the low-resolution series is provided, and show that the ranking between the methods changes in this setting. This is relevant to many practical use cases of MISR where the goal is to super-resolve a specific image of the series, i.e. the reference is known. The proposed PROBA-V-REF should better reflect the performance of the different methods for this reference-aware MISR problem.

Reducing methane emissions is essential to tackle climate change. Here, we address the problem of detecting large methane leaks by using hyperspectral data from the satellite Sentinel-5P. By sampling Sentinel-5P spectral data at fine scale, we detect methane absorption features in the shortwave infrared wavelength range (SWIR). Our method involves two separate steps: i) background subtraction and ii) detection of local maxima in the negative logarithmic spectrum of each pixel. In the first step, we remove the impact of the albedo using albedo maps and the impact of the atmosphere by using a principal component analysis (PCA) over a time series of past observations. In the second step, we count for each pixel the number of local maxima that correspond to a subset of local maxima in the methane absorption spectrum. This counting method allows us to set up a statistical a contrario test that controls the false alarm rate of our detections.

Cloud detection is a crucial step for automatic satellite image analysis. Some cloud detection methods exploit specially designed spectral bands, other base the detection on time series, or on the inter-band delay in push-broom satellites. Nevertheless many use cases occur where these methods do not apply. This paper describes a convolutional neural network for cloud detection in panchromatic and single-frame images. Only a per-image annotation is required, indicating which images contain clouds and which are cloud-free. Our experiments show that, in spite of using less information, the proposed method produces competitive results.

Unsupervised learning based on Contrastive Learning (CL) has attracted a lot of interest recently. This is due to excellent results on a variety of subsequent tasks (especially classification) on benchmark datasets (ImageNet, CIFAR-10, etc.) without the need of large quantities of labeled samples. This work explores the application of some of the most relevant CL techniques on a large unlabeled dataset of aerial images of building rooftops. The task that we want to solve is roof type classification using a much smaller labeled dataset. The main problem with this task is the strong dataset bias and class imbalance. This is caused by the abundance of certain types of roofs and the rarity of other types. Quantitative results show that this issue heavily affects the quality of learned representations, depending on the chosen CL technique.

We propose a self-supervised approach for training multi-frame video denoising networks. These networks predict each frame from a stack of frames around it. Our self-supervised approach benefits from the temporal consistency in the video by minimizing a loss that penalizes the difference between the predicted frame and a neighboring one, after aligning them using an optical flow. We use the proposed strategy to denoise a video contaminated with an unknown noise type, by fine-tuning a pre-trained denoising network on the noisy video. The proposed fine-tuning reaches and sometimes surpasses the performance of state-of-the-art networks trained with supervision. We demonstrate this by showing extensive results on video blind denoising of different synthetic and real noises. In addition, the proposed fine-tuning can be applied to any parameter that controls the denoising performance of the network. We show how this can be expoited to perform joint denoising and noise level estimation for heteroscedastic noise.

With the widespread application of convolutional neural networks (CNNs), the traditional model based denoising algorithms are now outperformed. However, CNNs face two problems. First, they are computationally demanding, which makes their deployment especially difficult for mobile terminals. Second, experimental evidence shows that CNNs often over-smooth regular textures present in images, in contrast to traditional non-local models. In this letter, we propose a solution to both issues by combining a nonlocal algorithm with a lightweight residual CNN. s solution gives full latitude to the advantages of both models. We apply this framework to two GPU implementations of classic nonlocal algorithms (NLM and BM3D) and observe a substantial gain in both cases, performing better than the state-of-the-art with low computational requirements. Our solution is between 10 and 20 times faster than CNNs with equivalent performance and attains higher PSNR. In addition the final method shows a notable gain on images containing complex textures like the ones of the MIT Moiré dataset.

The classic approach to image matching consists in the detection, description and matching of keypoints. In the description, the local information surrounding the keypoint is encoded. This locality enables affine invariant methods. Indeed, smooth deformations caused by viewpoint changes are well approximated by affine maps. Despite numerous efforts, affine invariant descriptors have remained elusive. This has led to the development of IMAS (Image Matching by Affine Simulation) methods that simulate viewpoint changes to attain the desired invariance. Yet, recent CNN-based methods seem to provide a way to learn affine invariant descriptors. Still, as a first contribution, we show that current CNN-based methods are far from the state-of-the-art performance provided by IMAS. This confirms that there is still room for improvement for learned methods. Second, we show that recent advances in affine patch normalization can be used to create adaptive IMAS methods that select their affine simulations depending on query and target images. The proposed methods are shown to attain a good compromise: on the one hand, they reach the performance of state-of-the-art IMAS methods but are faster; on the other hand, they perform significantly better than non-simulating methods, including recent ones. Source codes are available at https://rdguez-mariano.github.io/pages/adimas.

The classic approach to image matching consists in the detection, description and matching of keypoints. This defines a zero-order approximation of the mapping between two images, determined by corresponding point coordinates. But the patches around keypoints typically contain more information, which may be exploited to obtain a first-order approximation of the mapping, incorporating local affine maps between corresponding keypoints. In this work, we propose a LOCal Affine Transform Estimator (LOCATE) method based on neural networks. We show that LOCATE drastically improves the accuracy of local geometry estimation by tracking inverse maps. A second contribution on guided matching and refinement is also presented. The novelty here consists in the use of LOCATE to propose new SIFT-keypoint correspondences with precise locations, orientations and scales. Our experiments show that the precision gain provided by LOCATE does play an important role in applications such as guided matching. The third contribution of this paper consists in a modification to the RANSAC algorithm, that uses LOCATE to improve the homography estimation between a pair of images. These approaches outperform RANSAC for different choices of image descriptors and image datasets, and permit to increase the probability of success in identifying image pairs in challenging matching databases. The source codes are available at: https://rdguez-mariano.github.io/ pages/locate .

Modeling the processing chain that has produced a video is a difficult reverse engineering task, even when the camera is available. This makes model based video processing a still more complex task. In this paper we propose a fully blind video denoising method, with two versions off-line and on-line. This is achieved by fine-tuning a pre-trained AWGN denoising network to the video with a novel frame-to-frame training strategy. Our denoiser can be used without knowledge of the origin of the video or burst and the post-processing steps applied from the camera sensor. The on-line process only requires a couple of frames before achieving visually pleasing results for a wide range of perturbations. It nonetheless reaches state-of-the-art performance for standard Gaussian noise, and can be used off-line with still better performance.

Demosaicking and denoising are the first steps of any camera image processing pipeline and are key for obtaining high quality RGB images. A promising current research trend aims at solving these two problems jointly using convolutional neural networks. Due to the unavailability of ground truth data these networks cannot be currently trained using real RAW images. Instead, they resort to simulated data. In this paper we present a method to learn demosaicking directly from mosaicked images, without requiring ground truth RGB data. We apply this to learn joint demosaicking and denoising only from RAW images, thus enabling the use of real data. In addition we show that for this application fine-tuning a network to a specific burst improves the quality of restoration for both demosaicking and denoising.

Non-local patch-based methods were until recently state-of-the-art for image denoising but are now outperformed by convolutional neural networks (CNNs). Yet they are still the best ones for video denoising, as video redundancy is a key factor to attain high denoising performance. In this work we propose a novel video denoising CNN. Non-local self-similarity is incorporated into the network via a first non-trainable layer which finds for each patch in the input image its most similar patches in a 3D spatio-temporal search region centered at the target patch. The central values of these patches are then gathered in a feature vector which is assigned to each image pixel. This information is presented to a CNN which is trained to predict a clean image. The proposed architecture achieves state-of-the-art results. To the best of our knowledge, this is the first successful application of CNNs to video denoising.

The classic approach to image matching consists in the detection, description and matching of keypoints. The descriptor encodes the local information around the keypoint. An advantage of local approaches is that viewpoint deformations are well approximated by affine maps. This motivated the quest for affine invariant local descriptors. Despite numerous efforts, such descriptors remained elusive, ultimately resulting in the compromise of using viewpoint simulations to attain affine invariance. In this work we propose a CNN-based patch descriptor which captures affine invariance without the need for viewpoint simulations. This is achieved by training a neural network to associate similar vectorial representations to patches related by affine transformations. During matching, these vectors are compared very efficiently. The invariance to translation, rotation and scale is still obtained by the first stages of SIFT, which produce the keypoints. The proposed descriptor outperforms the state-of-the-art in retaining affine invariant properties.

New micro-satellite constellations enable unprecedented systematic monitoring applications thanks to their wide coverage and short revisit capabilities. However, the large volumes of images that they produce have uneven qualities, creating the need for automatic quality assessment methods. In this work, we quantify the sharpness of images from the PlanetScope constellation by estimating the blur kernel from each image. Once the kernel has been estimated, it is possible to compute an absolute measure of sharpness which allows to discard low quality images and deconvolve blurry images before any further processing. The method is fully blind and automatic, and since it does not require the knowledge of any satellite specifications it can be ported to other constellations.

The evolution of the antarctic ice cap is a subject of the utmost importance for the climate science. For this reason exploiting the historic archive of SPOT 5 HRS binocular stereo imagery over the Antarctica has gained interest. However, estimating surface models of Antarctica from optical satellite imagery is a challenging task. Different factors contribute to its difficulty: the reduced contrast of the snow, the abrupt changes in elevation, and the persistent cloud cover that is often indistinguishable from the snowy ground, or is translucent, or projects shadows on the ground; all of them hinder stereo matching. For these reasons the direct application of existing satellite stereo pipelines on these images often yields unsatisfactory results. In this study we explore strategies to address these problems and improve 3D modeling on these regions. We adapt the S2P [1] pipeline and incorporate a new multiscale strategy that allows to deal with incorrect geometry estimation due to clouds or lack of texture. In addition, integrating existing very low resolution DSM (1 km per pixel) of the Antarctica allows to further filter the model to produce a cleaner 3D model.

Most image deblurring methods assume an over-simplistic image formation model and as a result are sensitive to more realistic image degradations. We propose a novel variational framework, that explicitly handles pixel saturation, noise, quantization, as well as non-linear camera response function due to e.g., gamma correction. We show that accurately modeling a more realistic image acquisition pipeline leads to significant improvements, both in terms of image quality and PSNR. Furthermore, we show that incorporating the nonlinear response in both the data and the regularization terms of the proposed energy leads to a more detailed restoration than a naive inversion of the non-linear curve. The minimization of the proposed energy is performed using stochastic optimization. A dataset consisting of realistically degraded images is created in order to evaluate the method.

In this paper we reconsider the class of patch based denoising algorithms and observe that they underperform at lower image frequencies. We solve this problem by operating them within a multiscale structure. Our main observation is that denoising algorithms cannot be trusted with the restoration of high frequency details in the image. Indeed, since denoising algorithms must impose their image prior, the fine details are either smoothed or sharpened in the result. In any case the high frequency properties of the images are altered. This realization has a profound implication on the multiscale approaches, which assume that coarse scale restorations are better denoised and hence are replaced in the finer resolutions. This leads to frequency cut-off artifacts as the coarse restorations are pasted at higher resolutions. We start by studying this phenomenon on a simple Discrete Cosine Transform (DCT) pyramid, for which the artifacts resulting from this process are evident. We propose a simple solution consisting of a “conservative recomposition” of the scales that only retains the lower frequencies of each scale, with the obvious exception of the scale at the highest resolution. This soft fusion eliminates the ringing artifacts and attenuates staircasing artifacts and low frequency bumps. An added benefit of the DCT pyramid is that it allows one to maintain the white noise at the lower resolutions, hence it can be combined with any denoising algorithm without adaptation. This soft fusion recipe can be generalized to any other pyramid structure. We apply it to a Laplacian pyramid as an example. Our proposal merges and operates any denoising algorithm into a multiscale method, with improvements both in visual quality and Peak Signal to Noise Ratio (PSNR), and with little additional complexity. The method is demonstrated on several classic or state-of-the-art denoising algorithms.

International audience

This paper presents DA3D (Data Adaptive Dual Domain De-noising), a “last step denoising” method that takes as input a noisy image and as a guide the result of any state-of-the-art denoising algorithm. The method performs frequency domain shrinkage on shape and data-adaptive patches. Unlike other dual denoising methods, DA3D doesn't process all the image samples, which allows it to use large patches (64 × 64 pixels). The shape and data-adaptive patches are dynamically selected, effectively concentrating the computations on areas with more details, thus accelerating the process considerably. DA3D also reduces the staircasing artifacts sometimes present in smooth parts of the guide images. The effectiveness of DA3D is confirmed by extensive experimentation. DA3D improves the result of almost all state-of-the-art methods, and this improvement requires little additional computation time.

We consider the two-images stereo disparity problem favoring correctness of matches over density. We will deal with high resolution images which permit an accurate matching in textured zones, but which might present, as any stereo pair, ambiguities and occlusions. Global variational methods can estimate a dense map based on regularity assumptions about the disparity function. However, if these assumptions are incorrect this may lead to erroneous interpretations and mismatches. The availability of tristereo or even multiview stereo imagery permit us to combine the disparities from different pairs, allowing for a reliable densification not based on regularity assumptions. Local methods are suitable for this purpose since they permit us to check the validity of each match. The main disadvantages of local methods are the matching ambiguity and the failures of the fronto-parallel hypothesis (at places like discontinuities and slanted surfaces). We advocate, in this work, for the use of oriented windows in order to deal with slanted surfaces and discontinuities. Unlike adaptive support windows the oriented windows permit us to correctly estimate disparities on non-fronto-parallel surfaces. Several parameterless techniques for detecting mismatches are presented. The incorporation of these validation techniques in a coarse-to-fine multiwindow algorithm, allows us to obtain fairly dense results with few mismatches. An extensive comparison, including classical stereo pairs, high resolution satellite images, and images from the KITTI benchmark, illustrates the performance of the proposed method.

In this paper we study the problem of comparing two patches of images defined on Riemannian manifolds which in turn can be defined by each image domain with a suitable metric depending on the image. For that we single out one particular instance of a set of models defining image similarities that was earlier studied in [C. Ballester et al., Multiscale Model. Simul., 12 (2014), pp. 616--649], using an axiomatic approach that extended the classical Álvarez--Guichard--Lions--Morel work to the nonlocal case. Namely, we study a linear model to compare patches defined on two images in $\mathbb{R}^N$ endowed with some metric. Besides its genericity, this linear model is selected by its computational feasibility since it can be approximated leading to an algorithm that has the complexity of the usual patch comparison using a weighted Euclidean distance. Moreover, we propose and study some intrinsic metrics which we define in terms of affine covariant structure tensors and we discuss their properties. These tensors are defined for any point in the image and are intrinsically endowed with affine covariant neighborhoods. We also discuss the effect of discretization over the affine covariance properties of the tensors. We illustrate our theoretical results with numerical experiments.

Image stereo pairs obtained from pinhole cameras can be stereo-rectified, thus permitting to test and use the many standard stereo matching algorithms of the literature. Yet, it is well-known that pushbroom Earth observation satellites produce image pairs that are not stereo-rectifiable. Nevertheless, we show that by a new and adequate use of the satellite calibration data, one can perform a precise local stereo-rectification of large Earth images. Based on this we built a fully automatic 3D reconstruction chain for the new Pléiades Earth observation satellite. It produces 1/10 pixel accurate Earth image stereo pairs at a high resolution. Examples will be made available online to the computer vision community.

The current state-of-the-art non-local algorithms for image denoising have the tendency to remove many low contrast details. Frequency-based algorithms keep these details, but on the other hand many artifacts are introduced. Recently, the Dual Domain Image Denoising (DDID) method has been proposed to address this issue. While beating the state-of-the-art, this algorithm still causes strong frequency domain artifacts. This paper reviews DDID under a different light, allowing to understand their origin. The analysis leads to the development of NLDD, a new denoising algorithm that outperforms DDID, BM3D and other state-of-the-art algorithms. NLDD is also three times faster than DDID and easily parallelizable.

Modern Earth observation satellites are calibrated in such a way that a point on the ground can be located with an error of just a few pixels in the image domain. For many applications this error can be ignored, but this is not the case for stereo reconstruction, that requires sub-pixel accuracy. In this article we propose a method to correct this error. The method works by estimating local corrections that compensate the error relative to a reference image. The proposed method does not rely on ground control points, but only on the relative consistency of the image contents. We validate our method with Pléiades and WorldView-1 images on a representative set of geographic sites.

In this work we propose a non-local gradient-based energy for interpolating incomplete disparity maps. It represents an extension of the bilateral filter adapted to reconstruct locally planar disparity maps. We assume that we have at our disposal a reference image from which similarity weights can be computed. When the spatial extend of the weights tends to zero, the proposed model can be shown to converge to an energy involving second order derivatives, explaining thus its ability to obtain higher order interpolations. The proposed energy can be minimized by solving its Euler-Lagrange equation via an iteration of second order Poisson equations. By including an edge map our model permits also to recover depth discontinuities.

We present a method to compute the relative depth of moving objects in video sequences. The method relies on the fact that the boundary between two moving objects follows the movement of the object which is closest to the camera. Thus, the input of the method is a segmentation (to know the boundaries of objects) and an optical flow (to know the movement of the objects). The output of the method is a relative ordering of the neighboring segments. In fact, this output only provides a cue of the desired relative ordering, just like T-junctions typically provide a cue of the relative ordering of the objects around them. These cues can be used later as heuristics or as starting points for higher-level algorithms for image and video-processing.

We propose an interpolation method for sparse data that incorporates the geometric information of a reference image. The idea consists in defining for each sample a geodesic neighborhood and then fit a model (affine for instance) to interpolate at the current point. In the field of remote sensing for urban areas, two widely used techniques are laser range scanning (LIDAR) and stereo photogrammetry. Both techniques have a common drawback, for a variety of reasons the information they provide is sparse or incomplete. But in both cases it is fair to assume that a high resolution image of the scene is available, and we propose in this paper a diffusion algorithm that takes into account the geometry of the image u to refine the range data. This allows us to interpolate the data set while respecting the edges of u. The core of the algorithm is a fast method for computing geodesic distances between image points, which has been successfully applied to colorization by Yatziv et al. and supervised segmentation by Bai et al. The geodesic distance is used to find the set of points that are used to interpolate a piecewise affine model in the current sample. This first interpolation result is refined by merging the obtained affine patches using a greedy Mumford-Shah like algorithm. The output is a piecewise affine interplation of the data set that respects both the given data and the radiometric information provided by u.

The main purpose of this paper is to develop the mathematical analysis of anisotropic total variation problems with a degenerate metric and the computation of the associated Cheeger sets. We illustrate our analysis with the computation of Cheeger sets with respect to different anisotropic norms of relevance in applications to image processing. In particular, we describe the computation of global minima of geodesic active contour models, and we illustrate the use of Cheeger sets for the problem of edge linking.

Minimal surface regularization has been used in several applications ranging from stereo to image segmentation, sometimes hidden as a graph-cut discrete formulation, or as a strictly convex approximation to TV minimization. In this paper we consider a modified version of minimal surface regularization coupled with a robust data fitting term for interpolation purposes, where the corresponding evolution equation is constrained to diffuse only along the isophotes of a given image u and we design a convergent numerical scheme to accomplish this. To illustrate the usefulness of our approach, we apply this framework to the digital elevation model interpolation and to constrained vector probability diffusion. 1