This work addresses composed image retrieval in the context of domain conversion, where the content of a query image is retrieved in the domain specified by the query text. We show that a strong vision-language model provides sufficient descriptive power without additional training. The query image is mapped to the text input space using textual inversion. Unlike common practice that invert in the continuous space of text tokens, we use the discrete word space via a nearest-neighbor search in a text vocabulary. With this inversion, the image is softly mapped across the vocabulary and is made more robust using retrieval-based augmentation. Database images are retrieved by a weighted ensemble of text queries combining mapped words with the domain text. Our method outperforms prior art by a large margin on standard and newly introduced benchmarks. Code: https://github.com/NikosEfth/freedom

Unsupervised object-centric learning aims to decompose scenes into interpretable object entities, termed slots. Slot-based auto-encoders stand out as a prominent method for this task. Within them, crucial aspects include guiding the encoder to generate object-specific slots and ensuring the decoder utilizes them during reconstruction. This work introduces two novel techniques, (i) an attention-based self-training approach, which distills superior slot-based attention masks from the decoder to the encoder, enhancing object segmentation, and (ii) an innovative patch-order permutation strategy for autoregressive transformers that strengthens the role of slot vectors in reconstruction. The effectiveness of these strategies is showcased experimentally. The combined approach significantly surpasses prior slot-based autoencoder methods in unsupervised object segmentation, especially with complex real-world images. We provide the implementation code at https://github.com/gkakogeorgiou/spot.

During the last decade, free and open access to a variety of high spatial, temporal and spectral resolution earth observation (EO) data has brought a revolution in research and operational mapping applications. To this direction, the Prima MAGO (Mediterranean Water Management Solutions for Sustainable Agriculture Supplied by an Online Collaborative Platform) project provides novel solutions utilizing advanced remote sensing techniques, with the aim to enhance integrated water resources management for sustainable agriculture. In this paper, two software applications developed for mapping land cover/crop types and monitoring crop-water demand based on ESA Copernicus Sentinel-2 and Sentinel-3 data are presented. Implementation of the two MAGO software applications is demonstrated for two case studies: (i) mapping land cover in western Montpellier, France and (ii) monitoring crop water demand in the Cap Bon region, Tunisia.

In the realm of satellite-based marine monitoring, accurate cloud detection is crucial for ensuring the reliability of satellite imagery analysis. Addressing this challenge, our paper presents a comprehensive evaluation of four cloud detection algorithms (Fmask, SEN2COR, KappaMask, and S2Cloudless) based on MARIDA dataset, which includes annotations for marine debris, sea surface features, and cloud classes over marine environments in Sentinel-2 (S2) data. We introduce a new class, Thin Cloud, and significantly extend existing annotations to encompass diverse cloud characteristics. Our analysis employs an evaluation protocol with varying degrees of classification granularity, assessing algorithm performance across Cloud, Thin Cloud, Cloud Shadow, and Clear categories. The results reveal Fmask’s proficiency in binary Cloud/ Clear detection, while KappaMask demonstrates consistent performance across all scenarios, including complex Cloud Shadow distinctions. We also identify specific limitations of each algorithm, such as SEN2COR’s underper-formance in dense cloudy regions and Fmask’s challenges in identifying cloud shadows. Our findings provide valuable insights into the effectiveness of these algorithms in marine cloud detection, highlighting the need for improved accuracy in cloud masking techniques for marine monitoring systems.

Savannahs are vital ecosystems whose sustainability is endangered by the spread of woody plants. This research targets the accurate mapping of fractional woody cover (FWC) at the species level in a South African savannah, using EnMAP hyperspectral data. Field annotations were combined with very high-resolution multispectral drone data to produce land cover maps that included three woody species. The high-resolution labelled maps were then used to generate FWC samples for each woody species class at the 30-m spatial resolution of EnMAP. Four machine learning regression algorithms were tested for FWC mapping on dry season EnMAP imagery. The contribution of multitemporal information was also assessed by incorporating as additional regression features, spectro-temporal metrics from Sentinel-2 data of both the dry and wet seasons. The results demonstrated the suitability of our approach for accurately mapping FWC at the species level. The highest accuracy rates achieved from the combined EnMAP and Sentinel-2 experiments highlighted their synergistic potential for species-level vegetation mapping.

Snow, a crucial component of the cryosphere, significantly impacts global climate monitoring and has an important role in freshwater supply, hydropower energy and tourism. Advances in remote sensing technologies, particularly Interferometric Synthetic Aperture Radar (InSAR), enhance our understanding of snow processes. This study investigates the potential use of interferometric observables (phase, coherence, and phase closure) from a ground-based L-band SAR sensor to analyze snow dynamics. Time series data expanding one winter and one summer season are assessed with in-situ snow depth, snow water equivalent and air temperature observations. We found that by exploiting all three interferometric observables the identification of the start and the end of each snow season (wet snow, dry snow, no snow) can be performed more accurately. This study highlights the potential of L-band interferometric observables that are relevant for future L-band SAR missions.

Object tracking in satellite videos remains a complex endeavor in remote sensing due to the intricate and dynamic nature of satellite imagery. Existing state-of-the-art trackers in computer vision integrate sophisticated architectures, attention mechanisms, and multi-modal fusion to enhance tracking accuracy across diverse environments. However, the challenges posed by satellite imagery, such as background variations, atmospheric disturbances, and low-resolution object delineation, significantly impede the precision and reliability of traditional Single Object Tracking (SOT) techniques. Our study delves into these challenges and proposes prompt engineering methodologies, leveraging the Segment Anything Model (SAM) and TAPIR (Tracking Any Point with per-frame Initialization and temporal Refinement), to create a training-free point-based tracking method for small-scale objects on satellite videos. Experiments on the VISO dataset validate our strategy, marking a significant advancement in robust tracking solutions tailored for satellite imagery in remote sensing applications.

This work introduces composed image retrieval to remote sensing. It allows to query a large image archive by image examples alternated by a textual description, enriching the descriptive power over unimodal queries, either visual or textual. Various attributes can be modified by the textual part, such as shape, color, or context. A novel method fusing image-to-image and text-to-image similarity is introduced. We demonstrate that a vision-language model possesses sufficient descriptive power and no further learning step or training data are necessary. We present a new evaluation benchmark focused on color, context, density, existence, quantity, and shape modifications. Our work not only sets the state-of-the-art for this task, but also serves as a foundational step in addressing a gap in the field of remote sensing image retrieval. Code at: https://github.com/billpsomas/rscir.

Underwater environments provide significant opportunities for innovative research in various scientific fields and can have high economic interest in numerous commercial applications. To monitor diverse phenomena in an unattended manner, it is necessary to develop suitable observatory systems. In this work we describe the development of an autonomous submarine imaging system for bubble detection, the implementation of which addresses the challenges of deployment in extreme environments, such as those encountered in highly active underwater hydrothermal fields. To address the bubble detection problem, we train state-of-the-art object detection models using a manually collected dataset containing a large number of images in a real environment. Our models exhibit promising results both in terms of evaluation metrics and qualitative assessment.

We propose a training-free demosaicing method for multi-spectral filter array sensors by exploiting the concept of deep image prior. The proposed method does not require hard-to-obtain ground truth high resolution data corresponding to the bands of the filter array, and can thus be easily employed to arbitrary multispectral filter array configurations. Besides its training-free nature, the proposed method achieves improved performance with respect to state-of-the-art learning-based multispectral demosaicing methods.

The Harmonized LandSat-Sentinel dataset marks a significant stride in merging two esteemed Earth Observation satellite platforms, LandSat 8/9 and Sentinel-2. Despite the notable convergence, there’s a partial mismatch in features between their S30 and L30 products due to differing band specifications. To bridge this gap, our study employs cutting-edge generative neural networks. We utilize an Encoder-Decoder architecture, leveraging advanced SSL-pretrained backbones as feature encoders alongside a versatile CNN decoder. Training our model on the S30 HLS product, we capitalize on the precise harmonization of common bands, allowing direct application of our model to infer missing bands in the L30 product. We assess the accuracy of the four predicted bands using two evaluation approaches: comparing them to S30 products taken on the same day and analyzing their performance across a S30 product timeseries. Our findings demonstrate promising results, signaling a pathway towards achieving full feature parity between these two HLS products.

Real-time analysis of Martian craters is crucial for mission-critical operations, including safe landings and geological exploration. This work leverages the latest breakthroughs for on-the-edge crater detection aboard spacecraft. We rigorously benchmark several YOLO networks using a Mars craters dataset, analyzing their performance on embedded systems with a focus on optimization for low-power devices. We optimize this process for a new wave of cost-effective, commercial-off-the-shelf-based smaller satellites. Implementations on diverse platforms, including Google Coral Edge TPU, AMD Versal SoC VCK190, Nvidia Jetson Nano and Jetson AGX Orin, undergo a detailed trade-off analysis. Our findings identify optimal network-device pairings, enhancing the feasibility of crater detection on resource-constrained hardware and setting a new precedent for efficient and resilient extraterrestrial imaging. Code at: https://github.com/billpsomas/mars_crater_detection.

Convolutional networks and vision transformers have different forms of pairwise interactions, pooling across layers and pooling at the end of the network. Does the latter really need to be different? As a by-product of pooling, vision transformers provide spatial attention for free, but this is most often of low quality unless self-supervised, which is not well studied. Is supervision really the problem?In this work, we develop a generic pooling framework and then we formulate a number of existing methods as instantiations. By discussing the properties of each group of methods, we derive SimPool, a simple attention-based pooling mechanism as a replacement of the default one for both convolutional and transformer encoders. We find that, whether supervised or self-supervised, this improves performance on pre-training and downstream tasks and provides attention maps delineating object boundaries in all cases. One could thus call SimPool universal. To our knowledge, we are the first to obtain attention maps in supervised transformers of at least as good quality as self-supervised, without explicit losses or modifying the architecture. Code at: https://github.com/billpsomas/simpool.

Earth observation data of high spatial, spectral, and temporal resolution provide crucial information for monitoring efficiently agricultural practices and their environmental footprint. The per-parcel estimation of agricultural water requirements is a major challenge for water management, especially in regions vulnerable to climate change. To this end, in this study we employ satellite, meteorological and in-situ data for assessing the satellite derived evapotranspiration (ET) against the actual water-related stress measured in-situ through stomatal conductance. Sentinel-2 and Sentinel-3 data are employed and processed in an automated manner by the developed cloud-based software based on Sen-ET and TSEB-PT. Agrometeological and in-situ measurements of LAI and stomatal conductance have been collected and compared in several cotton, maize, and sunflower parcels in Greece. Overall, the qualitative and quantitative evaluation demonstrated a high R2(>0.65) and low MSE for all correlation models.

Savannahs are under threat from land degradation, not least due to woody vegetation encroachment. Here, we target the accurate high-resolution mapping of fractional woody vegetation cover in a South African savannah region, and assess the contribution of optical (Sentinel-2, S2), radar (Sentinel-1, S1) and ancillary data (bioclimatic and soils). Firstly, we created fractional woody cover (FCW) samples based on very high resolution satellite imagery and then we performed several regression experiments using different combinations of S1, S2 and auxiliary data temporal metrics for both dry and wet seasons. Highest regression rates of over 80% were documented for the all-data and all-season combined experiments, while S2 only models clearly outperformed the S1 ones. An interesting outcome from the results analysis was that appending ancillary bioclimatic and soils information to the S2-based experiments was proven to be more effective than the inclusion of S1 metrics.

During the last decades, the remote sensing community has gained access to a wide satellite imagery material, resulting in great progress on various applications. Most of these applications firstly require that the employed images are in the same coordinate system, without any registration errors that will deteriorate their performance. In this work we employ a multistep fully convolutional network in order to improve the image registration task. Moreover, we propose the relaxation of the registration constraints on the regions where changes have occurred. In this way, the model learns to keep the proper structures of objects that change in the multitemporal pair while at the same time provides dense deformations for the rest of the region. Our method is very efficient, fast and requires annotations for the regions of change only in training time. We conduct experiments on the very high resolution Attica VHR dataset comparing it with other deformable registration approaches from the literature. Our method outperforms all compared methods both quantitatively and qualitatively, setting up a foundation for further future research.

Savannahs cover 50% of the African continent and 20% of the global land surface. African savannahs are increasingly threatened by over-exploitation, deforestation, woody thickening and encroachment and, consequently, land degradation. In South Africa, savannah degradation is acute and accelerating, threatening the ecosystem services provided to some of the country's most vulnerable populations. Here, we devise a methodology for the accurate mapping and monitoring of the fraction of the woody component of savannah vegetation and apply it to the South African Northwest Province. Our approach involves the use of aerial photography for training and validation; the entire dry-season Landsat archive over the last three decades, and deep learning segmentation and classification techniques. Our results are able to identify areas of significant woody densification and encroachment, as well as areas with declining trends.

We explore the use of production Machine Learning (ML) frameworks for automatically building ML models for cloud-based services that exploit geospatial big data and value-added products. We combine two widely used production ML frameworks to hierarchically decompose the tasks involved with the fetching and preprocessing of the data as well as with model training, evaluation, and selection. We assess the usability, reproducibility and performance of the frameworks both qualitatively and quantitatively. We examine the challenging case of a cloud-based seabed mapping service that process multispecrtal multibeam echosounder data captured in different marine surveys, involving a number of data processing and machine learning tasks.

In this article, we present a deep multitask learning framework able to couple semantic segmentation and change detection using fully convolutional long short-term memory (LSTM) networks. In particular, we present a UNet-like architecture (L-UNet) that models the temporal relationship of spatial feature representations using integrated fully convolutional LSTM blocks on top of every encoding level. In this way, the network is able to capture the temporal relationship of spatial feature vectors in all encoding levels without the need to downsample or flatten them, forming an end-to-end trainable framework. Moreover, we further enrich the L-UNet architecture with an additional decoding branch that performs semantic segmentation on the available semantic categories that are presented in the different input dates, forming a multitask framework. Different loss quantities are also defined and combined together in a circular way to boost the overall performance. The developed methodology has been evaluated on three different data sets, i.e., the challenging bitemporal high-resolution Office National d'Etudes et de Recherches Aérospatiales (ONERA) Satellite Change Detection (OSCD) Sentinel-2 data set, the very high-resolution (VHR) multitemporal data set of the East Prefecture of Attica, Greece, and finally, the multitemporal VHR SpaceNet7 data set. Promising quantitative and qualitative results demonstrated that the synergy among the tasks can boost up the achieved performances. In particular, the proposed multitask framework contributed to a significant decrease in false-positive detections, with the F1 rate outperforming other state-of-the-art methods by at least 2.1% and 4.9% in the Attica VHR and SpaceNet7 data set cases, respectively. Our models and code can be found at https://github.com/mpapadomanolaki/multi-task-L-UNet.

Semantic segmentation, especially for very high-resolution satellite data, is one of the pillar problems in the remote sensing community. Lately, deep learning techniques are the ones that set the state-of-the-art for a number of benchmark datasets, however, there are still a lot of challenges that need to be addressed, especially in the case of limited annotations. To this end, in this paper, we propose a novel framework based on deep neural networks that is able to address concurrently semantic segmentation and image reconstruction in an end to end training. Under the proposed formulation, the image reconstruction acts as a regularization, constraining efficiently the solution in the entire image domain. This self-supervised component helps significantly the generalization of the network for the semantic segmentation, especially in cases of a low number of annotations. Experimental results and the performed quantitative evaluation on the publicly available ISPRS (WGIII/4) dataset indicate the great potential of the developed approach.

The advent of multitemporal high resolution data, like the Copernicus Sentinel-2, has enhanced significantly the potential of monitoring the earth's surface and environmental dynamics. In this paper, we present a novel deep learning framework for urban change detection which combines state-of-the-art fully convolutional networks (similar to U-Net) for feature representation and powerful recurrent networks (such as LSTMs) for temporal modeling. We report our results on the recently publicly available bi-temporal Onera Satellite Change Detection (OSCD) Sentinel-2 dataset, enhancing the temporal information with additional images of the same region on different dates. Moreover, we evaluate the performance of the recurrent networks as well as the use of the additional dates on the unseen test-set using an ensemble cross-validation strategy. All the developed models during the validation phase have scored an overall accuracy of more than 95%, while the use of LSTMs and further temporal information, boost the F1 rate of the change class by an additional 1.5%.

The detailed, accurate and frequent land cover and crop-type mapping emerge as essential for several scientific communities and geospatial applications. This paper presents a methodology for the semi-automatic production of land cover and crop type maps using a highly analytic nomenclature of more than 40 classes. An intensive manual annotation procedure was carried out for the production of reference data. A class nomenclature based on CORINE land cover Level-3 was employed along with several additional crop-type classes. Multitemporal surface reflectance Landsat-8 data for the year of 2016 were used for all classification experiments with a linear SVM classifier. Quantitative and qualitative evaluation highlighted the efficiency of the proposed approach achieving high accuracy rates. Further analysis on individual classes' performance highlighted the challenges in the proposed classification scheme as well as important outcomes regarding the spectral behavior of the considered categories.

Semantic segmentation is currently a mainstream method for addressing several remote sensing applications, achieving recently remarkable performance by employing deep learning techniques. In particular, this is the case for pixel-wise dense classification models in very high resolution remote sensing datasets. In this paper, we exploit the use of a relatively deep architecture based on repetitive downscale-upscale processes that had been previously employed for human pose estimation tasks. By integrating such a model, we are aiming to capture and extract low-level details, such as small objects, object boundaries and edges. Experimental results and quantitative evaluation has been performed on the publicly available ISPRS (WGIII4) benchmark dataset indicating the potential of the proposed approach.

Automatic and accurate detection of man-made objects, such as buildings, is one of the main problems that the remote sensing community has been focusing on for the last decades. In this paper, we propose a Conditional Random Field (CRF) formulation which is using edge/boundary localization priors towards accurate building detection. These edge priors have been integrated/fused with the classification scores from a deep learning Convolutional Neural Network (CNN) architecture under a single energy formulation. The validation of the developed methodology had been performed on the recently published SpaceNet dataset. Experimental results and quantitative evaluation, based on different accuracy statistics, indicate the great potential of the proposed approach.

Recent advances in photonics and imaging technology allow the development of cutting-edge, lightweight hyperspectral sensors, both push-broom/line-scanning and snapshot/frame. At the same time, emerging applications in robotics, food inspection, medicine and earth observation are posing critical challenges on real-time processing and computational efficiency, both in terms of accuracy and power consumption. In this direction, in the current paper, we accelerate hyperspectral processing kernels by utilizing FPGAs, i.e., Zynq-7000 SoC, to perform similarity-based matching of spectral signatures. We propose a custom HW architecture based on multi-level parallelization, modularity, and parametric VHDL coding, which allows for in-depth design space exploration and trade-off analysis. Depending on configuration, our implementation processes 22-107 Megapixels per second providing an acceleration of 40-355x vs Intel-i3 CPU and 360-104x vs the embedded ARM Cortex A9, whereas the overall detection quality ranges from 56% to 97% when evaluated with multiple objects and images of 285 spectral channels.

In this paper, a novel generic framework has been designed, developed and validated for addressing simultaneously the tasks of image registration, segmentation and change detection from multisensor, multiresolution, multitemporal satellite image pairs. Our approach models the inter-dependencies of variables through a higher order graph. The proposed formulation is modular with respect to the nature of images (various similarity metrics can be considered), the nature of deformations (arbitrary interpolation strategies), and the nature of segmentation likelihoods (various classification approaches can be employed). Inference of the proposed formulation is achieved through its mapping to an overparametrized pairwise graph which is then optimized using linear programming. Experimental results and the performed quantitative evaluation indicate the high potentials of the developed method.

In order to exploit operationally remote sensing data for agricultural applications efficient and automated methods are required towards the accurate detection of vegetation, crops and different crop varieties. To this end, an object-based classification framework has been developed and validated towards the detection of vineyards and the discrimination of vine varieties. Very high resolution satellite data were collected over four wine-growing regions in Greece during a three-year period, i.e., 2012 to 2014. A rule-based classification scheme based on fuzzy logic was employed in order to firstly detect the vine parcels in the pan-sharpened multispectral satellite images. Then the canopy of each parcel was detected and separated from the soil in-between the vine rows. The detection of the different vine varieties followed based on a supervised classification procedure and spectral features. The overall validation and quite promising experimental results indicated that a throughout sensitivity analysis can form efficient operational tools for variety-based data analysis in precision viticulture.

In this paper an automated vehicle detection and traffic density estimation algorithm has been developed and validated for very high resolution satellite video data. The algorithm is based on an adaptive background estimation procedure followed by a background subtraction at every video frame. The vehicle detection is performed through a further mathematical morphology and statistical analysis on the computed connected components. The traffic density has been estimated based on a lower resolution grid superimposed on the scene. In particular, at every subregion the number of the detected vehicles is calculated and the density is then estimated for the entire road network at every frame. The developed algorithm has been quantitatively evaluated. The quite promising results indicate the potentials of the proposed approach, while parallel GPU implementations can allow for real-time performance.

Spectral observations along the spectrum in many narrow spectral bands through hyperspectral imaging provides valuable information towards material and object recognition, which can be consider as a classification task. Most of the existing studies and research efforts are following the conventional pattern recognition paradigm, which is based on the construction of complex handcrafted features. However, it is rarely known which features are important for the problem at hand. In contrast to these approaches, we propose a deep learning based classification method that hierarchically constructs high-level features in an automated way. Our method exploits a Convolutional Neural Network to encode pixels' spectral and spatial information and a Multi-Layer Perceptron to conduct the classification task. Experimental results and quantitative validation on widely used datasets showcasing the potential of the developed approach for accurate hyperspectral data classification.

The automated man-made object detection and building extraction from single satellite images is, still, one of the most challenging tasks for various urban planning and monitoring engineering applications. To this end, in this paper we propose an automated building detection framework from very high resolution remote sensing data based on deep convolutional neural networks. The core of the developed method is based on a supervised classification procedure employing a very large training dataset. An MRF model is then responsible for obtaining the optimal labels regarding the detection of scene buildings. The experimental results and the performed quantitative validation indicate the quite promising potentials of the developed approach.

In this paper, a novel variational framework is introduced toward automatic 3-D building reconstruction from remote-sensing data. We consider a subset of building models that involve the footprint, their elevation, and the roof type. These models, under a certain hierarchical representation, describe the space of solutions and, under a fruitful synergy with an inferential procedure, recover the observed scene's geometry. Such an integrated approach is defined in a variational context, solves segmentation both in optical images and digital elevation maps, and allows multiple competing priors to determine their pose and 3-D geometry from the observed data. The very promising experimental results and the performed quantitative evaluation demonstrate the potentials of our approach.

In this paper, we introduce a variational framework towards automatic 3D building reconstruction from optical and Lidar data. Multiple 3D competing building priors are considered under a recognition-driven way. These models, under a certain hierarchical representation, describe the space of solutions and under a fruitful synergy with an inferential procedure recover the observed scene's geometry. Our formulation allows the cue with the higher spatial resolution to constrain properly the boundaries detection procedure ensuring, in this way, optimal results in terms of accuracy. Such an integrated approach is defined in a variational context, solves segmentation in both spaces, addresses fusion in a natural manner and allows multiple competing priors to determine the pose and 3D geometry from the observed data. Very promising experimental results demonstrate the potentials of our approach.

In this paper, a novel recognition-driven variational framework, toward multiple building extraction from aerial and satellite images, is introduced. To this end, competing shape priors are considered, and building extraction is addressed through an image segmentation approach that involves the use of a data-driven term constrained from the prior models. The proposed framework extends previous approaches toward the integration of multiple shape priors into the level-set segmentation. In particular, it estimates the number of buildings as well as their pose from the observed data. Therefore, it can address multiple building extraction from a single optical image, a highly demanding task of fundamental importance in various geoscience and remote-sensing applications. Furthermore, it can be easily extended to deal with other remote-sensing data through a simple modification of the image term. Very promising experimental results and the performed qualitative and quantitative evaluation demonstrate the potential of our approach.

In this paper, we propose a novel technique to represent and recover optical flow through free form deformations. Such a technique is based on representing the motion field using regular connected grids according to higher order polynomials, a compromise between dense motion estimation and parametric motion models. Optical flow is determined through the deformation of the grid - derived from the optimization of a cost function - and consequently of the underlying image structures towards satisfying the constant brightness constraint. Smoothness conditions are implicitly accounted for through the free form deformation approach. Promising results demonstrate the potentials of our approach.