Object detection is a crucial task in both computer vision and remote sensing. The performance of object detectors can vary across different modalities depending on lighting and weather conditions. To address these challenges, we propose a fusion module based on contrastive learning and Gaussian cross-channel attention, called Gaussian Contrastive Cross-Channel Fusion (G3CF). We integrate this module into a dual-YOLO architecture, forming YOLO-G3CF. The contrastive loss enforces similarity between the features sent to the detection head from both modality branches, as they should lead to the same detections. The Gaussian attention mechanism enables the model to fuse features in a higher-dimensional space, enhancing discriminative power. Extensive experiments on VEDAI, GeoImageNet, VTUAV-det, and FLIR demonstrate that G3CF improves detection performance, achieving a mAP increase of up to 6.64% over the best single-modality baselines and outperforming prior multimodal fusion methods. Regarding model complexity, our fusion method operates at a late stage, increasing the computational cost of single-modality YOLO by approximately 150% in terms of GFLOPs. For instance, YOLOv8 requires 52.84 GFLOPs, whereas YOLOv8-G3CF, due to its dual architecture and three G3CF modules, increases this to 131.22 GFLOPs. However, a single G3CF module requires only ~ 15 GFLOPs. Despite this overhead, our approach remains computationally less expensive than transformer-based models, e.g., ICAFusion requires 284.80 GFLOPs. Moreover, the proposed method still operates in real-time, achieving ~ 19 FPS on an NVIDIA RTX 2080. The code is available at https://github.com/abelmouhcine/YOLO-G3CF.

Forest monitoring using remotely sensed data is a central task in forestry and ecosystem studies. The long-term assessment of woody vegetation assists, for instance, in change detection and handling environmental hazards. Recently, France has made available its historical aerial images archive, covering the whole country extent. The public availability of such datasets expands the scope of long-term monitoring studies, such as forest monitoring. The main aim of this study is to investigate how the volume and the quality of the reference data influence the semantic segmentation of woody vegetation using historical grayscale aerial images in southwestern France. The two main contributions of this study are the provision of a rare 1942 binary dataset (woody vegetation vs. no woody vegetation) and the investigation of the effects of training data quality and quantity using a deep learning architecture.

Generally caused by extreme events, landslides cause severe landscape modifications and may endanger local population. It is important to be able to map them in order to better understand landscape evolution. 3D LiDAR point clouds (PCs) are a relevant choice compared to 2D imagery to directly sense ground shape modification under vegetated areas. Most of the studies propose to rely on rasterization of PCs, or multistep semi-automatic process with tedious manual results refinement in the 3D PCs. In this study, we aim at experimenting a deep learning method to directly extract landslide sources and deposits from raw 3D PCs. To this end, we train an Encoder Fusion SiamKPConv network, designed for 3D PCs change detection, for the specific task of landslides identification in PCs acquired before and after Kaikōura earthquake (New-Zealand). The experimental results (93.87% of accuracy) show the relevance of this model.

In this study, we tackle the challenge of identifying plant species from ultra high resolution (UHR) remote sensing images. Our approach involves introducing an RGB remote sensing dataset, characterized by millimeter-level spatial resolution, meticulously curated through several field expeditions across a mountainous region in France covering various landscapes. The task of plant species identification is framed as a semantic segmentation problem for its practical and efficient implementation across vast geographical areas. However, when dealing with segmentation masks, we confront instances where distinguishing boundaries between plant species and their background is challenging. We tackle this issue by introducing a fuzzy loss within the segmentation model. Instead of utilizing one-hot encoded ground truth (GT), our model incorporates Gaussian filter refined GT, introducing stochasticity during training. First experimental results obtained on both our UHR dataset and a public dataset are presented, showing the relevance of the proposed methodology, as well as the need for future improvement.

Regular patterns of vegetation are considered widespread landscapes, although their global extent has never been estimated. Among them, spotted landscapes are of particular interest in the context of climate change. Indeed, regularly spaced vegetation spots in semi-arid shrublands result from extreme resource depletion and prefigure catastrophic shift of the ecosystem to a homogeneous desert, while termite mounds also producing spotted landscapes were shown to increase robustness to climate change. Yet, their identification at large scale calls for automatic methods, for instance using the popular deep learning framework, able to cope with a vast amount of remote sensing data, e.g., optical satellite imagery. In this paper, we tackle this problem and benchmark some state-of-the-art deep networks on several landscapes and geographical areas. Despite the promising results we obtained, we found that more research is needed to be able to map automatically these earth mounds from space.

Change detection is an important task that rapidly identifies modified areas, particularly when multi-temporal data are concerned. In landscapes with a complex geometry (e.g., urban environment), vertical information is a very useful source of knowledge that highlights changes and classifies them into different categories. In this study, we focus on change segmentation using raw three-dimensional (3D) point clouds (PCs) directly to avoid any information loss due to the rasterization processes. While deep learning has recently proven its effectiveness for this particular task by encoding the information through Siamese networks, we investigate herein the idea of also using change information in the early steps of deep networks. To do this, we first propose to provide a Siamese KPConv state-of-the-art (SoTA) network with hand-crafted features, especially a change-related one, which improves the mean of the Intersection over Union (IoU) over the classes of change by 4.70%. Considering that a major improvement is obtained due to the change-related feature, we then propose three new architectures to address 3D PC change segmentation: OneConvFusion, Triplet KPConv, and Encoder Fusion SiamKPConv. All these networks consider the change information in the early steps and outperform the SoTA methods. In particular, Encoder Fusion SiamKPConv overtakes the SoTA approaches by more than 5% of the mean of the IoU over the classes of change, emphasizing the value of having the network focus on change information for the change detection task. The code is available at https://github.com/IdeGelis/torch-points3d-SiamKPConvVariants.

We propose a new model of Variational Autoencoder (VAE) for Anomaly Detection (AD) with improved modeling power. More precisely, we introduce a VAE model with a Gaussian Random Field (GRF) prior, namely VAE-GRF, which generalizes the classical VAE model. We show that, under some assumptions, the VAE-GRF largely outperforms the traditional VAE and some other probabilistic models developed for AD. Our experimental results suggest that the VAE-GRF could be used as a relevant VAE baseline in place of the traditional VAE with very limited additional computational cost. We provide competitive results on the public MVTec benchmark dataset for visual inspection, as well as on the public Livestock dataset dedicated to the task of unsupervised animal detection from aerial images.

Object detection in remote sensing is a crucial computer vision task that has seen significant advancements with deep learning techniques. However, most existing works in this area focus on the use of generic object detection and do not leverage the potential of multimodal data fusion. In this paper, we present a comparison of methods for multimodal object detection in remote sensing, survey available multimodal datasets suitable for evaluation, and discuss future directions.

New satellite constellations allow the acquisition of high temporal and spatial resolution images at any point on the Earth. These data, assembled in the form of satellite image time series (SITS), are an important source of information for monitoring the evolution of the Earth’s surface. Deep learning is one of the most promising solutions for the automatic analysis of large volumes of data acquired by new generations of satellites. However, these techniques often only exploit temporal or spatial structures. To take advantage of the temporal and spatial complementarity of the data without computational burden, we use graph-based modeling in combination with deep learning. In particular, we propose a comparison of five graph neural networks applied to SITS. The results highlight the efficiency of graph models in understanding the spatio-temporal context of regions, which might lead to a better classification compared to attribute-based methods.

This paper is concerned with the change detection issue in remote sensing images. This problem is not trivial since the notion of change depends on the application. Moreover, classical supervised deep learning methods have to deal with the limited amount of labelled data available. Based on existing deep learning techniques that exploit unsupervised clustering to assign labels to entire images, we adapt them to the change detection problem by using siamese backbones and extracting pixel-wise results. As fully unsupervised experiments lead to unstable results, we suggest "low supervision" strategy composed of a warm-up stage with few labeled data able to drive the following unsupervised learning through reliable solutions. Preliminary experiments show reliable change maps.

The computer vision community is increasingly interested in exploring hyperbolic space for image representation, as hyperbolic approaches have demonstrated outstanding results in efficiently representing data with an underlying hierarchy. This interest arises from the intrinsic hierarchical nature among images. However, despite the hierarchical nature of remote sensing (RS) images, the investigation of hyperbolic spaces within the RS community has been relatively limited. The objective of this study is therefore to examine the relevance of hyperbolic embeddings of RS data, focusing on scene embedding. Using a Variational Auto-Encoder, we project the data into a hyperbolic latent space while ensuring numerical stability with a feature clipping technique. Experiments conducted on the NWPU-RESISC45 image dataset demonstrate the superiority of hyperbolic embeddings over the Euclidean counterparts in a classification task. Our study highlights the potential of operating in hyperbolic space as a promising approach for embedding RS data.

This study aims at recovering above-ground biomass information from ultra-high resolution UAV RGB-NIR orthophotos. We focus on a realistic scenario where a limited number of training samples for a landscape with heterogeneous herbaceous vegetation is given. Consequently, we explore different machine learning methods explicitly addressing the limitations of small training samples and compare their predictions quantitatively and qualitatively. Our results show that random forest models perform similarly well to deep learning models. While simpler machine learning models may, therefore, still be preferable, our study also points the way to promising architectures and regularisation techniques for deep learning approaches. Beyond vegetation cover, accurate regression of other variables, including vegetation height, volume and biomass remains a difficult task regardless of the model choice.

This paper studies a reconstruction-based approach for weakly-supervised animal detection from aerial images in marine environments. Such an approach leverages an anomaly detection framework that computes metrics directly on the input space, enhancing interpretability and anomaly localization compared to feature embedding methods. Building upon the success of Vector-Quantized Variational Autoencoders in anomaly detection on computer vision datasets, we adapt them to the marine animal detection domain and address the challenge of handling noisy data. To evaluate our approach, we compare it with existing methods in the context of marine animal detection from aerial image data. Experiments conducted on two dedicated datasets demonstrate the superior performance of the proposed method over recent studies in the literature. Our framework offers improved interpretability and localization of anomalies, providing valuable insights for monitoring marine ecosystems and mitigating the impact of human activities on marine animals.

Choosing how to encode a real-world problem as a machine learning task is an important design decision in machine learning. The task of glacier calving front modeling has often been approached as a semantic segmentation task. Recent studies have shown that combining segmentation with edge detection can improve the accuracy of calving front detectors. Building on this observation, we completely rephrase the task as a contour tracing problem and propose a model for explicit contour detection that does not incorporate any dense predictions as intermediate steps. The proposed approach, called “Charting Outlines by Recurrent Adaptation” (COBRA), combines Convolutional Neural Networks (CNNs) for feature extraction and active contour models for the delineation. By training and evaluating on several large-scale datasets of Greenland’s outlet glaciers, we show that this approach indeed outperforms the aforementioned methods based on segmentation and edge-detection. Finally, we demonstrate that explicit contour detection has benefits over pixel-wise methods when quantifying the models’ prediction uncertainties. The project page containing the code and animated model predictions can be found at https://khdlr.github.io/COBRA/.

Most existing Multi-Object Tracking methods consider detection and re-identification as two distinct steps. As a result, the re-identification cannot leverage object location and is only based on appearance, thus leading to ID merges when dealing with highly similar objects. The few works that combine detection and re-identification still generate an appearance descriptor for similarity computation. However, since the detection task conflicts with the tracking task, the network privileges the former and generates similar descriptors for objects of the same class, especially when class instances have a strong visual similarity. Besides, when using a motion model or a motion prediction recurrent neural network to delimit the search area and overcome the problem of ID merges, the rise of uncertainty occurring when those models are not updated often leads to ID switches. In this paper, we tackle these issues and propose to use the same model for detection and re-identification by leveraging attention between features of two frames. By doing so, the network can make motion predictions without providing any appearance descriptor and without computing any learned similarity, thus eliminating the need for any motion prediction model and making the tracking trainable end-to-end. Our experimental results support our main contributions and show that our fully DeepSORT significantly reduces the number of ID switches and merges, even when using non-class-agnostic non-maximum suppression. Besides, our model is more resistant to variations in time lapses between two images, leading to improved tracking results.

Anomaly Detection (AD) is an important research topic, with very diverse applications such as industrial defect detection, medical diagnosis, fraud detection, intrusion detection, etc. Within the last few years, deep learning-based methods have become the standard approach for AD. In many practical cases, the anomalies are unknown in advance. Therefore, most of challenging AD problems need to be addressed in an unsupervised or weakly supervised framework. In this context, deep generative models are widely used, in particular Variational Autoencoder (VAE) models. VAEs have been extended to Vector-Quantized VAEs (VQ-VAEs), a model increasingly popular because of its versatility enabled by the discrete latent space. We present for the first time a robust approach which takes advantage of the inner metrics of VQ-VAEs for AD. We show that the distance between the output of the encoder and the codebook vectors of a VQ-VAE provides a valuable information which can be used to localize the anomalies. In our approach, this metric complements a reconstruction-based metric to improve AD results. We compare our model with state-of-the-art AD models on three standards datasets, including the MVTec, UCSD-Ped1 and CIFAR-10 datasets. Experiments show that the proposed method yields high competitive results.

We present a deep active contour model for detecting and delineating glacier calving fronts from satellite imagery. Contrary to existing deep learning-based calving front detectors, our model does not perform an intermediate segmentation or pixel-wise edge detection, but instead directly predicts the contour parametrized by a fixed number of vertices. The model works by first deriving feature maps from an input image, and then updating an initial contour in an iterative fashion. Evaluating on the CALFIN dataset, which maps calving fronts in Greenland, our model outperforms existing approaches. Code for the experiments and animated predictions can be found at https://github.com/khdlr/deep-acm

Despite its robust performance under various illumination conditions, multispectral scene analysis has not been widely deployed due to two strong practical limitations: 1) thermal cameras, especially high-resolution ones are much more expensive than conventional visible cameras; 2) the most commonly adopted multispectral architectures, two-stream neural networks, nearly double the inference time of a regular mono-spectral model which makes them impractical in embedded environments. In this work, we aim to tackle these two limitations by proposing a novel knowledge distillation framework named Modality Distillation (MD). The proposed framework distils the knowledge from a high thermal resolution two-stream network with feature-level fusion to a low thermal resolution one-stream network with image-level fusion. We show on different multispectral scene analysis benchmarks that our method can effectively allow the use of low-resolution thermal sensors with more compact one-stream networks.

Deep learning, together with the availability of large amounts of data, has transformed the way we process Earth observation (EO) tasks, such as land cover mapping or image registration. Yet, today, new models are needed to push further the revolution and enable new possibilities. This work focuses on a recent framework for generative modeling and explores its applicability to the EO images. The framework learns an energy-based model (EBM) to estimate the underlying joint distribution of the data and the categories, obtaining a neural network that is able to classify and synthesize images. On these two tasks, we show that EBMs reach comparable or better performances than convolutional networks on various public EO datasets and that they are naturally adapted to semisupervised settings, with very few labeled data. Moreover, models of this kind allow us to address high-potential applications, such as out-of-distribution analysis and land cover mapping with confidence estimation.

Data from multiple sensors provide independent and complementary information, which may improve the robustness and reliability of scene analysis applications. While there exist many large-scale labelled benchmarks acquired by a single sensor, collecting labelled multi-sensor data is more expensive and time-consuming. In this work, we explore the construction of an accurate multispectral (here, visible & thermal cameras) scene analysis system with minimal annotation efforts via an active learning strategy based on the cross-modality prediction inconsistency. Experiments on multispectral datasets and vision tasks demonstrate the effectiveness of our method. In particular, with only 10% of labelled data on KAIST multispectral pedestrian detection dataset, we obtain comparable performance as other fully supervised State-of-the-Art methods.

Deep learning has changed unbelievably the processing of Earth Observation tasks such as land cover mapping or image registration. Yet, today new models are needed to push further the revolution and enable new possibilities. We propose a new framework for generative modelling of Earth Observation images. It learns an energy-based model to estimate the underlying distribution of the data while jointly training a deep neural network for classification. On the varied image types of the EuroSAT benchmark, we show this model obtains classification results on par with state-of-the-art and moreover allows us to tackle a wide range of high-potential applications: image synthesis, out-of-distribution testing for domain adaptation, and image completion or denoising.

Despite the intense development of deep neural networks for computer vision, and especially semantic segmentation, their application to Earth Observation data remains usually below accuracy requirements brought by real-life scenarios. Even if well-known deep learning methods produce excellent results, they tend to be over-confident and cannot assess how relevant their predictions are. In this work, a Bayesian deep learning method, based on Monte Carlo Dropout, is proposed to tackle semantic segmentation of aerial and satellite images. Bayesian deep learning can provide both a semantic segmentation and uncertainty maps. Based on the popular U-Net architecture, our model achieves semantic segmentation with high accuracy, e.g. F1-score and overall accuracy respectively reaching 90.84% and 93.22% on a public standard dataset. Uncertainty maps, also derived from our model, show a strong interest in qualitative evaluation of the segmentation and in the improvement of the database.

According to the United Nations, 70% of earth population is going to live in cities by 2050. Given this fast urban evolution, urban monitoring is a key process to qualify sustainable development. Vertical changes need to be assessed, and various methods for 3D change detection have been published. However, there is no common quantitative benchmark assessing their performance in urban areas yet. In this paper, we aim to fill this gap and introduce a simulation tool to generate synthetic 3D point cloud data in a well-controlled scenario. These data are then used to compare qualitatively and quantitatively representative 3D change detection methods for urban areas. These methods are based on distance computation (DSMd, C2C, M3C2), traditional machine learning (RF with stability feature) and deep learning (Feed Forward and Siamese networks). We distinguish between binary and multi-class classification of changes at different levels (3D points, 2D pixels, and 2D patches). While deep neural networks have led to numerous success in remote sensing, we show that they do not systematically outperform more simple methods for 3D change detection. Besides, the existing networks are limited to 2D patches while outputs at the pixel or point scale are more attractive.

Multispectral image pairs can provide complementary visual information, making pedestrian detection systems more robust and reliable. To benefit from both RGB and thermal IR modalities, we introduce a novel attentive multispectral feature fusion approach. Under the guidance of the inter- and intra-modality attention modules, our deep learning architecture learns to dynamically weigh and fuse the multispectral features. Experiments on two public multi-spectral object detection datasets demonstrate that the proposed approach significantly improves the detection accuracy at a low computation cost.

Multispectral images (e.g. visible and infrared) may be particularly useful when detecting objects with the same model in different environments (e.g. day/night outdoor scenes). To effectively use the different spectra, the main technical problem resides in the information fusion process. In this paper, we propose a new halfway feature fusion method for neural networks that leverages the complementary/consistency balance existing in multispectral features by adding to the network architecture, a particular module that cyclically fuses and refines each spectral feature. We evaluate the effectiveness of our fusion method on two challenging multispectral datasets for object detection. Our results show that implementing our Cyclic Fuse-and-Refine module in any network improves the performance on both datasets compared to other state-of-the-art multispectral object detection methods.

Morphological hierarchies form a popular framework aiming at emphasizing the multiscale structure of digital image by performing an unsupervised spatial partitioning of the data. These hierarchies have been recently extended to cope with image sequences, and different strategies have been proposed to allow their construction from spatio-temporal data. In this paper, we compare these hierarchical representation strategies for image sequences according to their structural properties. We introduce a projection method to make these representations comparable. Furthermore, we extend one of these recent strategies in order to obtain more efficient hierarchical representations for image sequences. Experiments were conducted on both synthetic and real datasets, the latter being made of satellite image time series. We show that building one hierarchy by using spatial and temporal information together is more efficient comparing to other existing strategies.

In this paper, we investigate and improve the use of a super-resolution approach to benefit the detection of small objects from aerial and satellite remote sensing images. The main idea is to focus the super-resolution on target objects within the training phase. Such a technique requires a reduced number of network layers depending on the desired scale factor and the reduced size of the target objects. The learning of our super-resolution network is performed using deep residual blocks integrated in a Wasserstein Generative adversarial network. Then, detection task is performed by exploiting two state-of-the-art detectors including Faster-RCNN and YOLOv3. Experiments were conducted on small vehicle detection from both aerial and satellite images from the VEDAI and xView data sets. Results showed that object-focused super-resolution improves the detection performance and facilitates the transfer learning from one data set to another.

Detection of new infrastructures (commercial, logistics, industrial or residential) from satellite images constitutes a proven method to investigate and follow economic and urban growth. The level of activities or exploitation of these sites may be hardly determined by building inspection, but could be inferred from vehicle presence from nearby streets and parking lots. We present in this paper two deep learning-based models for vehicle counting from optical satellite images coming from the Pleiades sensor at 50-cm spatial resolution. Both segmentation (Tiramisu) and detection (YOLO, You Only Look Once) architectures were investigated. These networks were adapted, trained and validated on a data set including 87k vehicles, annotated using an interactive semi-automatic tool developed by the authors. Experimental results show that both segmentation and detection models could achieve a precision rate higher than 85 % with a recall rate also high (76.4 % and 71.9 % for Tiramisu and YOLO respectively).

LiDAR point clouds are receiving a growing interest in remote sensing as they provide rich information to be used independently or together with optical data sources, such as aerial imagery. However, their nonstructured and sparse nature make them difficult to handle, conversely to raw imagery for which many efficient tools are available. To overcome this specific nature of LiDAR point clouds, the standard approach relies on converting the point cloud into a digital elevation model, represented as a 2-D raster. Such a raster can then be used similarly as optical images, e.g., with 2-D convolutional neural networks (CNNs) for semantic segmentation. In this letter, we show that LiDAR point clouds provide more information than only the digital elevation model and that considering alternative rasterization strategies helps to achieve better semantic segmentation results. We illustrate our findings on the IEEE Data Fusion Contest (DFC) 2018 data set.

We propose to jointly learn multi-view geometry and warping between views of the same object instances for robust cross-view object detection. What makes multi-view object instance detection difficult are strong changes in viewpoint, lighting conditions, high similarity of neighbouring objects, and strong variability in scale. By turning object detection and instance re-identification in different views into a joint learning task, we are able to incorporate both image appearance and geometric soft constraints into a single, multi-view detection process that is learnable end-to-end. We validate our method on a new, large data set of street-level panoramas of urban objects and show superior performance compared to various baselines. Our contribution is threefold: a large-scale, publicly available data set for multi-view instance detection and re-identification; an annotation tool custom-tailored for multi-view instance detection; and a novel, holistic multi-view instance detection and re-identification method that jointly models geometry and appearance across views.

This paper deals with strategies for LiDAR data analysis. While a large majority of studies first rasterize 3D point clouds onto regular 2D grids and then use 2D image processing tools for characterizing data, our work rather suggests to keep as long as possible the 3D structure by computing features on 3D data and rasterize later in the process. By this way, the vertical component is still taken into account. In practice, a voxelization step of raw data is performed in order to exploit mathematical tools defined on regular volumes. More precisely, we focus on attribute profiles that have been shown to be very efficient features to characterize remote sensing scenes. They require the computation of an underlying hierarchical structure (through a Max-Tree). Experimental results obtained on urban LiDAR data classification support the performances of this strategy compared with an early rasterization process.

Morphological attribute profiles have been one of the most effective image features for spatial-spectral classification of remote sensing images during the last decade. The motivation of this paper is to extend attribute profiles to satellite image time series, i.e. taking into account the temporal information. We introduce different approaches and report their performances for land cover mapping. Experiments are conducted on a Sentinel-2 dataset considering well-established supervised classification methods that are Random Forest and Support Vector Machines.

Nowadays, remote sensing technologies greatly ease environmental assessment using aerial images. Such data are most often analyzed by a manual operator, leading to costly and non scalable solutions. In the fields of both machine learning and image processing, many algorithms have been developed to fasten and automate this complex task. Their main common assumption is the need to have prior ground truth available. However, for field experts or engineers, manually labeling the objects requires a time-consuming and tedious process. Restating the labeling issue as a binary classification one, we propose a method to assist the costly annotation task by introducing an active learning process, considering a query-by-group strategy. Assuming that a comprehensive context may be required to assist the annotator with the labeling task of a single instance, the labels of all the instances of an image are indeed queried. A score based on instances distribution is defined to rank the images for annotation and an appropriate retraining step is derived to simultaneously reduce the interaction cost and improve the classifier performances at each iteration. A numerical study on real images is conducted to assess the algorithm performances. It highlights promising results regarding the classification rate along with the chosen re-training strategy and the number of interactions with the user.

This work addresses the scarcity of annotated hyperspectral data required to train deep neural networks. Especially, we investigate generative adversarial networks and their application to the synthesis of consistent labeled spectra. By training such networks on public datasets, we show that these models are not only able to capture the underlying distribution, but also to generate genuine-looking and physically plausible spectra. Moreover, we experimentally validate that the synthetic samples can be used as an effective data augmentation strategy. We validate our approach on several public hyperspectral datasets using a variety of deep classifiers.

Over the recent years, there has been an increasing interest in large-scale classification of remote sensing images. In this context, the Inria Aerial Image Labeling Benchmark has been released online in December 2016. In this paper, we discuss the outcomes of the first year of the benchmark contest, which consisted in dense labeling of aerial images into building / not building classes, covering areas of five cities not present in the training set. We present four methods with the highest numerical accuracies, all four being convolutional neural network approaches. It is remarkable that three of these methods use the U-net architecture, which has thus proven to become a new standard in image dense labeling.

In the context of the Spot World Heritage (SWH) initiative, the long term archive of Spot 1 to 5 satellite data will be reprocessed. This initiative gives the opportunity to increase Spot data quality and production efficiency with the introduction of new state-of-the-art processing methods. In this context, we consider the Spot 5 Supermode processing chain and the introduction of a new denoising method. Our objective is to propose the best denoising method in terms of efficiency and accuracy for the Spot 5 Supermode data production. We report two experimentations that lead to the choice of the Non Local Bayes (NLB) denoising method; it is fast, accurate and remains stable in terms of efficiency and quality for different landscapes and image sizes. We also introduce two optimized versions of the NLB algorithm that increase the computation efficiency by a factor up to 4.

The motivation of this paper is to conduct a comparative study on remote sensing image classification using the morphological attribute profiles (APs) and feature profiles (FPs) generated from different types of tree structures. Over the past few years, APs have been among the most effective methods to model the image's spatial and contextual information. Recently, a novel extension of APs called FPs has been proposed by replacing pixel gray-levels with some statistical and geometrical features when forming the output profiles. FPs have been proved to be more efficient than the standard APs when generated from component trees (max-tree and min-tree). In this work, we investigate their performance on the inclusion tree (tree of shapes) and partition trees (alpha tree and omega tree). Experimental results from both panchromatic and hyperspectral images again confirm the efficiency of FPs compared to APs.

In this paper, we adapt the Faster-RCNN framework for the detection of underground buried objects (i.e. hyperbola reflections) in B-scan ground penetrating radar (GPR) images. Due to the lack of real data for training, we propose to incorporate more simulated radargrams generated from different configurations using the gprMax toolbox. Our designed CNN is first pre-trained on the grayscale Cifar-10 database. Then, the Faster-RCNN framework based on the pre-trained CNN is trained and fine-tuned on both real and simulated GPR data. Preliminary detection results show that the proposed technique can provide significant improvements compared to classical computer vision methods and hence becomes quite promising to deal with this kind of specific GPR data even with few training samples.

Morphological attribute profiles (APs) have thus far been proven effective for remote sensing image classification by several research studies. However, recent studies have shown that a direct application of APs to highly textured and structured images, especially in very high-resolution (VHR) optical imagery, may be insufficient. Some solutions have been proposed to deal with this issue, such as to extract the local histograms and the local features of AP images [histogram-based APs (HAPs) and local feature-based APs (LFAP), respectively], or to combine APs with different textural features. In this letter, we review these approaches and then propose a novel strategy, which directly generates APs on some derived textural features instead of separately combining them. Experimental results from both natural textures and VHR optical remotely sensed images show that the proposed approach can produce superior classification performance than the standard APs, HAPs, LFAPs, as well as the classical combination of APs with textural features.

This paper introduces an extension of morphological attribute profiles (APs) by extracting their local features. The so-called local feature-based APs (LFAPs) are expected to provide a better characterization of each APs' filtered pixel (i.e., APs' sample) within its neighborhood, and hence better deal with local texture information from the image content. In this paper, LFAPs are constructed by extracting some simple first-order statistical features of the local patch around each APs' sample such as mean, standard deviation, and range. Then, the final feature vector characterizing each image pixel is formed by combining all local features extracted from APs of that pixel. In addition, since the self-dual APs (SDAPs) have been proved to outperform the APs in recent years, a similar process will be applied to form the local feature-based SDAPs (LFSDAPs). In order to evaluate the effectiveness of LFAPs and LFSDAPs, supervised classification using both the random forest and the support vector machine classifiers is performed on the very high resolution Reykjavik image as well as the hyperspectral Pavia University data. Experimental results show that LFAPs (respectively, LFSDAPs) can considerably improve the classification accuracy of the standard APs (respectively, SDAPs) and the recently proposed histogram-based APs.

With the rapid development of Remote Sensing acquisition techniques, there is a need to scale and improve processing tools to cope with the observed increase of both data volume and richness. Among popular techniques in remote sensing, Deep Learning gains increasing interest but depends on the quality of the training data. Therefore, this paper presents recent Deep Learning approaches for fine or coarse land cover semantic segmentation estimation. Various 2D architectures are tested and a new 3D model is introduced in order to jointly process the spatial and spectral dimensions of the data. Such a set of networks enables the comparison of the different spectral fusion schemes. Besides, we also assess the use of a “noisy ground truth” (i.e. outdated and low spatial resolution labels) for training and testing the networks.

The present paper introduces an extension of attribute profiles (APs) by extracting their local features. The so-called local feature-based attribute profiles (LFAPs) are expected to provide a better characterization of each APs' filtered pixel (i.e. APs' sample) within its neighborhood, hence better deal with local texture information from the image's content. In this work, LFAP is constructed by extracting some simple first-order statistical features of the local patch around each APs' sample such as mean, standard deviation, range, etc. Then, the final feature vector characterizing each image pixel is formed by combining all local features extracted from APs of that pixel. In order to evaluate the effectiveness of the proposed technique, supervised classification using Random Forest classifier is performed on the VHR panchromatic Reykjavik image. Experimental results show that LFAPs can considerably improve the classification accuracy of the standard APs and the recently proposed histogram-based APs.

This paper investigates the problem of glacier flow estimation using Synthetic Aperture Radar (SAR) image data. Our motivation is to exploit a weighted graph model constructed from characteristic points (i.e. keypoints) to measure the displacement vectors located at their positions. In fact, characteristic points are capable of capturing the image's radiometric and contextual information. Then, by encoding their interaction and inter-connection, a graph model is able to characterize both intensity and geometry information from the image content, which is relevant for texture tracking task. In this work, we employ a graph-based similarity measure to track the local texture information around each keypoint in order to figure out its correspondence from the other image and calculate the associated displacement. The proposed approach is tested and evaluated using high resolution TerraSAR-X images acquired from the Argentiere Glacier located in the French Alps. Our preliminary experimental results show the algorithm's capacity to provide a fast and reliable estimation of glacier flows, especially over highly textured and structured regions.

In this paper, we discuss and review how combined multiview imagery from satellite to street level can benefit scene analysis. Numerous works exist that merge information from remote sensing and images acquired from the ground for tasks such as object detection, robots guidance, or scene understanding. What makes the combination of overhead and street-level images challenging are the strongly varying viewpoints, the different scales of the images, their illuminations and sensor modality, and time of acquisition. Direct (dense) matching of images on a per-pixel basis is thus often impossible, and one has to resort to alternative strategies that will be discussed in this paper. For such purpose, we review recent works that attempt to combine images taken from the ground and overhead views for purposes like scene registration, reconstruction, or classification. After the theoretical review, we present three recent methods to showcase the interest and potential impact of such fusion on real applications (change detection, image orientation, and tree cataloging), whose logic can then be reused to extend the use of ground-based images in remote sensing andvice versa. Through this review, we advocate that cross fertilization between remote sensing, computer vision, and machine learning is very valuable to make the best of geographic data available from Earth observation sensors and ground imagery. Despite its challenges, we believe that integrating these complementary data sources will lead to major breakthroughs in Big GeoData. It will open new perspectives for this exciting and emerging field.

Designing an effective criterion to select a subset of features is a challenging problem for hyperspectral image classification. In this paper, we develop a feature selection method to select a subset of class discriminant features for hyperspectral image classification. First, we propose a new class separability measure based on the surrogate kernel and Hilbert Schmidt independence criterion in the reproducing kernel Hilbert space. Second, we employ the proposed class separability measure as an objective function and we model the feature selection problem as a continuous optimization problem using LASSO optimization framework. The combination of the class separability measure and the LASSO model allows selecting the subset of features that increases the class separability information and also avoids a computationally intensive subset search strategy. Experiments conducted with three hyperspectral data sets and different experimental settings show that our proposed method increases the classification accuracy and outperforms the state-of-the-art methods.

In this paper, we investigate the impact of segmentation algorithms as a preprocessing step for classification of remote sensing images in a deep learning framework. Especially, we address the issue of segmenting the image into regions to be classified using pre-trained deep neural networks as feature extractors for an SVM-based classifier. An efficient segmentation as a preprocessing step helps learning by adding a spatially-coherent structure to the data. Therefore, we compare algorithms producing superpixels with more traditional remote sensing segmentation algorithms and measure the variation in terms of classification accuracy. We establish that superpixel algorithms allow for a better classification accuracy as a homogenous and compact segmentation favors better generalization of the training samples.

Generating accurate and robust classification maps from hyperspectral imagery (HSI) depends on the choice of the classifiers and input data sources. Choosing the appropriate classifier for a problem at hand is a tedious task. Multiple classifier system (MCS) combines the relative merits of various classifiers to generate robust classification maps. However, the presence of inaccurate classifiers may degrade the classification performance of MCS. In this paper, we propose an unsupervised classifier selection strategy to select an appropriate subset of accurate classifiers for the multiple classifier combination from a large pool of classifiers. The experimental results with two HSI show that the proposed classifier selection method overcomes the impact of inaccurate classifiers and significantly increases the classification accuracy.

Morphological attribute profiles are among the most prominent spectral-spatial pixel description methods. They are efficient, effective, and highly customizable multiscale tools based on hierarchical representations of a scalar input image. Their application to multivariate images in general and hyperspectral images in particular has been so far conducted using the marginal strategy, i.e., by processing each image band (eventually obtained through a dimension reduction technique) independently. In this paper, we investigate the alternative vector strategy, which consists in processing the available image bands simultaneously. The vector strategy is based on a vector-ordering relation that leads to the computation of a single max and min tree per hyperspectral data set, from which attribute profiles can then be computed as usual. We explore known vector-ordering relations for constructing such max trees and, subsequently, vector attribute profiles and introduce a combination of marginal and vector strategies. We provide an experimental comparison of these approaches in the context of hyperspectral classification with common data sets, where the proposed approach outperforms the widely used marginal strategy.

This article explores the application of a tree-based feature extraction algorithm for the widely-used MSER features, and proposes a Tree of Shapes based detector of Maximally Stable Regions. Changing an underlying component tree in the algorithm allows considering alternative properties and pixel orderings for extracting the Maximally Stable Regions. Differences introduced to the region structure with changing the underlying tree are discussed, as well as the spatial organization of the detected regions imposed by using a self-dual image representation for detection. Performance evaluation is carried out on a standard matching benchmark in terms of repeat ability and matching score under different image transformations, as well as in a large scale image retrieval setup, measuring Mean Average Precision. The proposed de-scriptor is compared to the standard MSER implementation as well as a tree-based MSER implementation, achieving competitive results in the matching setup and outperforming the baseline MSER in the retrieval experiments.

We propose a local region descriptor based on connected pattern spectra, and combined with normalized central moments. The descriptors are calculated for MSER regions of the image, and their performance compared against SIFT. The MSER regions were chosen because they can be efficiently selected by constructing a max-tree, a structure used to calculate both descriptors and region moments. Experiments on the UCID database show an improvement over SIFT in two out of five experimental setups, and comparable performance in two other experiments. The new descriptors are only half the size of SIFT, resulting in 4 times faster query times when performing exact search on descriptor index built from 262 images.

Building detection is a challenging issue that requires efficient solutions in many operational contexts. Hierarchical image representation through tree structures is known for its compliance with such requirements for fast methods in remote sensing. In this paper, we address the building detection problem using an underlying hierarchical image model, and rely solely on height information coming from digital surface models (DSM). The proposed method is thus very efficient and allows for interactive rule-based detection of buildings and other land use / land cover classes. Preliminary results on a standard dataset from ISPRS are provided and illustrate the relevance of such an approach.

Despite the popularity of mathematical morphology with remote sensing image analysis, its application to hyperspectral data remains problematic. The issue stems from the need to impose a complete lattice structure on the multi-dimensional pixel value space, that requires a vector ordering. In this article, we introduce such a supervised ordering relation, which conversely to its alternatives, has been designed to be image-specific and exploits the spectral purity of pixels. The practical interest of the resulting multivariate morphological operators is validated through classification experiments where it achieves state-of-the-art performance.

A new classification technique, PerTurbo, has been investigated in the context on hyperspectral remote sensing images context. In this framework, each class is characterised by its Laplace-Beltrami operator, then approximated by the spectrum of K(S), whose terms are derived from the Gaussian kernel. The method is very simple, easy to implement and involves few parameters to tune. It also allows the definition of a simple multi-class strategy, and, as a class-wise classification method, the addition of a new class does not requires the re-training of the pre-existing class models. We conducted experiments on two datasets: results for Pavia Centre dataset are encouraging, while results obtained on Pavia University show that SVM clearly outperforms PerTurbo. Nevertheless, we believe that this difference comes from a bad parametrization of the algorithm (for which we used a rule of thumb, contrarily to the SVM procedure which was fully optimized). Hence, a systematic search for the optimal value of the parameter would improve the results. Moreover, there are several other possible improvements coming from the fields of regularization methods of dimensionality reduction techniques which let us think that this first experiment is promising. In the near future, we also plan to investigate rules leading to a better choice of s. We are also interested in studying the behavior of PerTurbo when the classes are heterogeneous with only few training samples available, or when the classes in the training set are highly unbalanced.

Connections in image processing are an important notion that describes how pixels can be grouped together according to their spatial relationships and/or their gray-level values. In recent years, several works were devoted to the development of new theories of connections among which hyperconnection (h-connection) is a very promising notion. This paper addresses two major issues of this theory. First, we propose a new axiomatic that ensures that every h-connection generates decompositions that are consistent for image processing and, more precisely, for the design of h-connected filters. Second, we develop a general framework to represent the decomposition of an image into h-connections as a tree that corresponds to the generalization of the connected component tree. Such trees are indeed an efficient and intuitive way to design attribute filters or to perform detection tasks based on qualitative or quantitative attributes. These theoretical developments are applied to a particular fuzzy h-connection, and we test this new framework on several classical applications in image processing, i.e., segmentation, connected filtering, and document image binarization. The experiments confirm the suitability of the proposed approach: It is robust to noise, and it provides an efficient framework to design selective filters.

The impact of PET on radiation therapy is held back by poor methods of defining functional volumes of interest. Many new software tools are being proposed for contouring target volumes but the different approaches are not adequately compared and their accuracy is poorly evaluated due to the illdefinition of ground truth. This paper compares the largest cohort to date of established, emerging and proposed PET contouring methods, in terms of accuracy and variability. We emphasise spatial accuracy and present a new metric that addresses the lack of unique ground truth. 30 methods are used at 13 different institutions to contour functional VOIs in clinical PET/CT and a custom-built PET phantom representing typical problems in image guided radiotherapy. Contouring methods are grouped according to algorithmic type, level of interactivity and how they exploit structural information in hybrid images. Experiments reveal benefits of high levels of user interaction, as well as simultaneous visualisation of CT images and PET gradients to guide interactive procedures. Method-wise evaluation identifies the danger of over-automation and the value of prior knowledge built into an algorithm.

International audience

In this paper, we investigate the possibilities offered by the extension of the connected component trees (cc-trees) to multivariate images. We propose a general framework for image processing using the cc-tree based on the lattice theory and we discuss the possible applications depending on the properties of the underlying ordered set. This theoretical reflexion is illustrated by two applications in multispectral astronomical imaging: source separation and object detection.

Placed within the context of content-based image retrieval, we study in this paper the potential of morphological operators as far as color description is concerned, a booming field to which the morphological framework, however, has only recently started to be applied. More precisely, we present three morphology-based approaches, one making use of granulometries independently computed for each subquantized color and two employing the principle of multiresolution histograms for describing color, using respectively morphological levelings and watersheds. These new morphological color descriptors are subsequently compared against known alternatives in a series of experiments, the results of which assert the practical interest of the proposed methods.

In this article, we focus on the problem of caption detection in video sequences. Contrary to most of existing approaches based on a single detector followed by an ad hoc and costly post-processing, we have decided to consider several detectors and to merge their results in order to combine advantages of each one. First we made a study of captions in video sequences to determine how they are represented in images and to identify their main features (color constancy and background contrast, edge density and regularity, temporal persistence). Based on these features, we then select or define the appropriate detectors and we compare several fusion strategies which can be involved. The logical process we have followed and the satisfying results we have obtained let us validate our contribution.

In this paper, we are dealing with color object tracking. We propose to use hidden Markov models in a different way as classical approaches. Indeed, we use these mathematical tools to model the object in the spatial domain rather than in the temporal domain. Besides in order to manage multidimensional (color) data, multidimensional hidden Markov models are involved. Object learning step is performed using the GHOSP algorithm whereas object tracking step is done by approximate object position prediction and then precise object position localisation. This last step can be seen as an object recognition problem and will be solved using a method based on the forward algorithm.

In this paper, we present a new automatic segmentation method for magnetic resonance images. The aim of this segmentation is to divide the brain into homogeneous regions and to detect the presence of tumors. Our method is divided into two parts. First, we make a pre-segmentation to extract the brain from the head. Then, a second segmentation is done inside the brain. Several techniques are combined like anisotropic filtering or stochastic model-based segmentation during the two processes. The paper describes the main features of the method, and gives some segmentation results.