Abstract The Urban Heat Island (UHI) effect, defined as a significant increase in temperature in urban environments compared to surrounding areas, is difficult to study in real cities using sensor data (satellites or in‐situ stations) due to their coarse spatial and temporal resolution. Among the factors contributing to this effect are the properties of urban materials, which differ from those in rural areas. To analyze their individual impact and to test new material configurations, a high‐resolution simulation at the city scale is required. Estimating the current materials used in a city, including those on building facades, is also challenging. We propose HeatMat, an approach to analyze at high resolution the individual impact of urban materials on the UHI effect in a real city, relying only on open data. We estimate building materials using street‐view images and a pre‐trained vision‐language model (VLM) to supplement existing OpenStreetMap data, which describes the 2D geometry and features of buildings. We further encode this information into a set of 2D maps that represent the city's vertical structure and material characteristics. These maps serve as inputs for our 2.5D simulator, which models coupled heat transfers and enables random‐access surface temperature estimation at multiple resolutions, reaching a 20x speedup compared to an equivalent simulation in 3D.

We present TerraMind, the first any-to-any generative, multimodal foundation model for Earth observation (EO). Unlike other multimodal models, TerraMind is pretrained on dual-scale representations combining both token-level and pixel-level data across modalities. On a token level, TerraMind encodes high-level contextual information to learn cross-modal relationships, while on a pixel level, TerraMind leverages fine-grained representations to capture critical spatial nuances. We pretrained TerraMind on nine geospatial modalities of a global, large-scale dataset. In this paper, we demonstrate that (i) TerraMind's dual-scale early fusion approach unlocks a range of zero-shot and few-shot applications for Earth observation, (ii) TerraMind introduces "Thinking-in-Modalities" (TiM) -- the capability of generating additional artificial data during finetuning and inference to improve the model output -- and (iii) TerraMind achieves beyond state-of-the-art performance in community-standard benchmarks for EO like PANGAEA. The pretraining dataset, the model weights, and our code are open-sourced under a permissive license.

Performing accurate confidence quantification and assessment in pixel-wise regression tasks, which are downstream applications of AI Foundation Models for Earth Observation (EO), is important for deep neural networks to predict their failures, improve their performance and enhance their capabilities in real-world applications, for their practical deployment. For pixel-wise regression tasks, specifically utilizing remote sensing data from satellite imagery in EO Foundation Models, confidence quantification is a critical challenge. The focus of this research is on developing a Foundation Model using EO satellite data that computes and assigns a confidence metric alongside regression outputs to improve the reliability and interpretability of predictions generated by deep neural networks. To this end, we develop, train and evaluate the proposed Confidence-Aware Regression Estimation (CARE) Foundation Model. Our model CARE computes and assigns confidence to regression results as downstream tasks of a Foundation Model for EO data, and performs a confidence-aware self-corrective learning method for the low-confidence regions. We evaluate the model CARE, and experimental results on multi-spectral data from the Copernicus Sentinel-2 constellation to estimate the building density (i.e. monitoring urban growth), show that the proposed method can be successfully applied to important regression problems in EO. We also show that our model CARE outperforms other methods.

Confidence assessments of semantic segmentation algorithms are important. Ideally, models should have the ability to predict in advance whether their output is likely to be incorrect. Assessing the confidence levels of model predictions in Earth observation (EO) classification is essential, as it can enhance semantic segmentation performance and help prevent further exploitation of the results in the case of erroneous prediction. The model we developed, Confidence Assessment for enhanced Semantic segmentation (CAS), evaluates confidence at both the segment and pixel levels, providing both labels and confidence scores as output. Our model, CAS, identifies segments with incorrectly predicted labels using the proposed combined confidence metric, refines the model, and enhances its performance. This work has significant applications, particularly in evaluating EO Foundation Models on semantic segmentation downstream tasks, such as land-cover classification using Sentinel-2 satellite data. The evaluation results show that this strategy is effective and that the proposed model CAS outperforms other baseline models.

Data from satellites or aerial vehicles are most of the times unlabelled. Annotating such data accurately is difficult, requires expertise, and is costly in terms of time. Even if Earth Observation (EO) data were correctly labelled, labels might change over time. Learning from unlabelled data within a semi-supervised learning framework for segmentation of aerial images is challenging. In this paper, we develop a new model for semantic segmentation of unlabelled images, the Non-annotated Earth Observation Semantic Segmentation (NEOS) model. NEOS performs domain adaptation as the target domain does not have ground truth masks. The distribution inconsistencies between the target and source domains are due to differences in acquisition scenes, environment conditions, sensors, and times. Our model aligns the learned representations of the different domains to make them coincide. The evaluation results show that it is successful and outperforms other models for semantic segmentation of unlabelled data.

Massive amounts of unlabelled data are captured by Earth Observation (EO) satellites, with the Sentinel-2 constellation generating 1.6TB of data daily. This makes Remote Sensing a data-rich domain well suited to Machine Learning (ML) solutions. However, a bottleneck in applying ML models to EO is the lack of annotated data as annotation is a labour-intensive and costly process. As a result, research in this domain has focused on Self-Supervised Learning and Foundation Model approaches. This paper addresses the need to evaluate different Foundation Models on a fair and uniform benchmark by introducing the PhilEO Bench, a novel evaluation framework for EO Foundation Models. The framework comprises of a testbed and a novel 400GB Sentinel-2 dataset containing labels for three downstream tasks, building density estimation, road segmentation, and land cover classification. We present experiments using our framework evaluating different Foundation Models, including Prithvi and SatMAE, at multiple n-shots and convergence rates.

Nowadays the accurate geo-localization of ground-view images has an important role across domains as diverse as journalism, forensics analysis, transports, and Earth Observation. This work addresses the problem of matching a query ground-view image with the corresponding satellite image without GPS data. This is done by comparing the features from a ground-view image and a satellite one, innovatively leveraging the corresponding latter’s segmentation mask through a three-stream Siamese-like network. The proposed method, Semantic Align Net (SAN), focuses on limited Field-of-View (FoV) and ground panorama images (images with a FoV of 360°). The novelty lies in the fusion of satellite images in combination with their segmentation masks, aimed at ensuring that the model can extract useful features and focus on the significant parts of the images. This work shows how SAN through semantic analysis of images improves the performance on the unlabelled CVUSA dataset for all the tested FoVs.

Baggage screening is important in security-critical applications in airports for detecting threats, including firearms and parts of them. Existing approaches underperform to recognise prohibited objects that are disassembled, especially when learning from limited data and from images produced by different scanners with multi-view orientations. To address such limitations, in this paper, we develop the Similarity Learning X-ray screening (SLX) model for accurate and robust firearm component detection in cluttered scenes. We evaluate SLX on the X-ray Image Library (XIL) dataset that the UK Government has provided us with, for this research. SLX is based on a contrastive similarity learning approach combined with Out-of-Distribution (OoD) detection/ anomaly detection using a deep discriminative model, ResNet-152, for detecting and classifying forbidden items. The evaluation of SLX on the XIL dataset shows that it is effective, beneficial for detecting firearms and their parts, and outperforms other baseline models, on average, by approximately 12 points in accuracy.

AI-enabled systems in security, autonomous systems, safety, and healthcare do not only need to effectively detect Out-of-Distribution (OoD) samples, but also to recognize Objects of Concern (OoC), e.g. multiple thorax diseases, efficiently with few-shots. Detecting OoD samples is crucial, because reporting an out-of-domain input as abnormal is better than falsely classifying it. Data samples, during inference, are not confined to a finite labelled set, and thus closed-set approaches are limiting, as they misclassify OoD inputs, and they may assign them high prediction confidence. Furthermore, although anomaly detection is possible, recognizing new OoC fast using only few-shot samples remains challenging. There is a lack of methodologies for joint anomaly detection and few-shot OoC classification. Our contribution is the development of a framework for joint few-shot OoC detection and classification and anomaly detection in the unknown previously-unseen, in the wild, environment, which is known as Open-World Recognition (OWR). We propose a novel methodology, the data distribution boundary Contrastive Training Recognition (CTR) classifier for few-shot OWR. CTR takes advantage of labels and classes to learn the normal (and few-shot abnormal) data better, to more accurately detect OoD. The proposed model: (i) reduces failures to detect anomalies in health- and safety-critical applications for avoiding unfavourable consequences, (ii) decreases false alarms, and (iii) improves performance overall. Our framework differs from existing approaches because: (a) anomaly and OoC detection are combined, which has several benefits, including improved OoD performance, (b) the performance, accuracy, and robustness of OoD and few-shot OoC detection are improved by strengthening the estimation of the normal class distribution at the boundary of its support, self-generating samples and setting them as abnormal, and (c) the knowledge base of models is also augmented by learning class-incrementally, alleviating forgetting. CTR outperforms baselines in several settings, including on the SVHN, CIFAR-FS, and BSCD-FS ChestX and ISIC datasets.

Detecting objects which we are interested in, Objects of Concern (OoC), is nowadays attracting attention. In aviation and transport, it is important to robustly detect OoC in security images. OoC are rare, differ from typical samples, and may be unknown during training. Most OoC detection methods need to be trained on large datasets to achieve good performance and have limited real-world generalization ability. A large variety of samples is needed, and it is expensive to collect and label large datasets due to the rarity of OoC. To address such limitations, we propose the negative REtraining with Few-shots Generative Adversarial Network (REFGAN) for detecting OoC. REFGAN aims at automatically identifying OoC by learning from Objects of No Concern (OoNC) and OoC. Our methodology comprises learning a prior using OoNC, and few-shot adaptation using the Few-Shot OoC (FSOoC). We propose a methodology to robustly perform few-shot adaptive detection of OoC using GANs and learned distribution boundaries. The evaluation of REFGAN on the Baggage SIXray dataset shows that when FSOoC are used, our model outperforms the prior improving OoC detection, and outperforms recent benchmarks by approximately 6.3% in mean values. REFGAN using few-shots of 80 samples shows a robust comparable performance compared to REFGAN using all the samples for retraining and model adaptation. REFGAN can detect unknown OoC and its evaluation on SIXray and CIFAR-10 shows robustness against the number of few-shot samples of OoC. REFGAN on CIFAR-10 outperforms benchmarks by approximately 16% using few-shots of 80 and of 10 samples.

Generative models, such as Generative Adversarial Networks (GANs), have been used for unsupervised anomaly detection. While performance keeps improving, several limitations exist particularly attributed to difficulties at capturing multimodal supports and to the ability to approximate the underlying distribution closer to the tails, i.e. the boundary of the distribution's support. This paper proposes an approach that attempts to alleviate such shortcomings. We propose an invertible-residual-network-based model, the Boundary of Distribution Support Generator (BDSG). GANs generally do not guarantee the existence of a probability distribution and here, we use the recently developed Invertible Residual Network (IResNet) and Residual Flow (ResFlow), for density estimation. These models have not yet been used for anomaly detection. We leverage IResNet and ResFlow for Out-of-Distribution (OoD) sample detection and for sample generation on the boundary using a compound loss function that forces the samples to lie on the boundary. The BDSG addresses non-convex support, disjoint components, and multimodal distributions. Results on synthetic data and data from multimodal distributions, such as MNIST and CIFAR-10, demonstrate competitive performance compared to methods from the literature.

We describe a monaural speech enhancement algorithm based on modulation-domain Kalman filtering to blindly track the time-frequency log-magnitude spectra of speech and reverberation. We propose an adaptive algorithm that performs blind joint denoising and dereverberation, while accounting for the inter-frame speech dynamics, by estimating the posterior distribution of the speech log-magnitude spectrum given the log-magnitude spectrum of the noisy reverberant speech. The Kalman filter update step models the non-linear relations between the speech, noise, and reverberation log spectra. The Kalman filtering algorithm uses a signal model that takes into account the reverberation parameters of the reverberation time T60and the direct-to-reverberant energy ratio (DRR) and also estimates and tracks T60and the DRR in every frequency bin to improve the estimation of the speech log spectrum. The proposed algorithm is evaluated in terms of speech quality, speech intelligibility, and dereverberation performance for a range of reverberation parameters and reverberant speech to noise ratios, in different noises, and is also compared to competing denoising and dereverberation techniques. Experimental results using noisy reverberant speech demonstrate the effectiveness of the enhancement algorithm.

We present a speech enhancement algorithm that performs modulation-domain Kalman filtering to track the speech phase using circular statistics, along with the spectral log-amplitudes of speech and noise. In the proposed algorithm, the speech phase posterior is used to create an enhanced speech phase spectrum for the signal reconstruction of speech. The Kalman filter prediction step separately models the temporal inter-frame correlation of the speech and noise spectral log-amplitudes and of the speech phase, while the Kalman filter update step models their nonlinear relations under the assumption that speech and noise add in the complex short-time Fourier transform domain. The phase-sensitive enhancement algorithm is evaluated with speech quality and intelligibility metrics, using a variety of noise types over a range of SNRs. Instrumental measures predict that tracking the speech log-spectrum and phase with modulation-domain Kalman filtering leads to consistent improvements in speech quality, over both conventional enhancement algorithms and other algorithms that perform modulation-domain Kalman filtering.