Advances in Earth observation (EO) foundation models have unlocked the potential of big satellite data to learn generic representations from space, benefiting a wide range of downstream applications crucial to our planet. However, most existing efforts remain limited to fixed spectral sensors, focus solely on the Earth's surface, and overlook valuable metadata beyond imagery. In this work, we take a step towards next-generation EO foundation models with three key components: 1) Copernicus-Pretrain, a massive-scale pretraining dataset that integrates 18.7M aligned images from all major Copernicus Sentinel missions, spanning from the Earth's surface to its atmosphere; 2) Copernicus-FM, a unified foundation model capable of processing any spectral or non-spectral sensor modality using extended dynamic hypernetworks and flexible metadata encoding; and 3) Copernicus-Bench, a systematic evaluation benchmark with 15 hierarchical downstream tasks ranging from preprocessing to specialized applications for each Sentinel mission. Our dataset, model, and benchmark greatly improve the scalability, versatility, and multimodal adaptability of EO foundation models, while also creating new opportunities to connect EO, weather, and climate research. Codes, datasets and models are available at https://github.com/zhu-xlab/Copernicus-FM.

We explore the potential of large-scale noisily labeled data to enhance feature learning by pretraining semantic segmentation models within a multimodal framework for geospatial applications. We propose a novel cross-modal sample selection (CromSS) method, a weakly supervised pretraining strategy designed to improve feature representations through cross-modal consistency and noise mitigation techniques. Unlike conventional pretraining approaches, CromSS exploits massive amounts of noisy and easy-to-come-by labels for improved feature learning beneficial to semantic segmentation tasks. We investigate middle and late fusion strategies to optimize the multimodal pretraining architecture design. We also introduce a cross-modal sample selection module to mitigate the adverse effects of label noise, which employs a cross-modal entangling strategy to refine the estimated confidence masks within each modality to guide the sampling process. Additionally, we introduce a spatial–temporal label smoothing technique to counteract overconfidence for enhanced robustness against noisy labels. To validate our approach, we assembled the multimodal dataset, NoLDO-S12, which consists of a large-scale noisy label subset from Google’s Dynamic World (DW) dataset for pretraining and two downstream subsets with high-quality labels from Google DW and OpenStreetMap (OSM) for transfer learning. Experimental results on two downstream tasks and the publicly available DFC2020 dataset demonstrate that when effectively utilized, the low-cost noisy labels can significantly enhance feature learning for segmentation tasks. The data, codes, and pretrained weights are freely available athttps://github.com/zhu-xlab/CromSS.

Compared to supervised deep learning, self-supervision provides remote sensing a tool to reduce the amount of exact, human-crafted geospatial annotations. While image-level information for unsupervised pretraining efficiently works for various classification downstream tasks, the performance on pixel-level semantic segmentation lags behind in terms of model accuracy. On the contrary, many easily available label sources (e.g., automatic labeling tools and land cover land use products) exist, which can provide a large amount of noisy labels for segmentation model training. In this work, we propose to exploit noisy semantic segmentation maps for model pretraining. Our experiments provide insights on robustness per network layer. The transfer learning settings test the cases when the pretrained encoders are fine-tuned for different label classes and decoders. The results from two datasets indicate the effectiveness of task-specific supervised pretraining with noisy labels. Our findings pave new avenues to improved model accuracy and novel pretraining strategies for efficient remote sensing image segmentation.

Local climate zones (LCZs) established a standard classification system to categorize the landscape universe for improved urban climate studies. Existing LCZ mapping is guided by human interaction with geographic information systems (GIS) or modelled from remote sensing (RS) data. GIS-based methods do not scale to large areas. However, RS-based methods leverage machine learning techniques to automatize LCZ classification from RS. Yet, RS-based methods require huge amounts of manual labels for training. We propose a novel LCZ mapping framework, termed AutoLCZ, to extract the LCZ classification parameters from high-resolution RS modalities. We study the definition of numerical rules designed to mimic the LCZ definitions. Those rules model geometric and surface cover parameters from LiDAR data. Correspondingly, we enable LCZ classification from RS data in a GIS-based scheme. The proposed AutoLCZ method has potential to reduce the human labor to acquire accurate metadata. At the same time, AutoLCZ sheds light on the physical interpretability of RS-based methods. In a proof-of-concept for New York City (NYC) we leverage airborne LiDAR surveys to model four LCZ parameters to distinguish eight LCZ types. The results indicate the potential of AutoLCZ as a promising avenue for large-scale LCZ mapping from RS data.

While the volume of remote sensing data is increasing daily, deep learning in Earth Observation faces lack of accurate annotations for supervised optimization. Crowdsourcing projects such as OpenStreetMap distribute the annotation load to their community. However, such annotation inevitably generates noise due to insufficient control of the label quality, lack of annotators, frequent changes of the Earth’s surface as a result of natural disasters and urban development, among many other factors. We presentAdaptively trIggered Online Object-wise correction (AIO2)to address annotation noise induced by incomplete label sets. AIO2 features anAdaptive Correction Trigger (ACT)module that avoids label correction when the model training under- or overfits, and anOnline Object-wise Correction (O2C)methodology that employs spatial information for automated label modification. AIO2 utilizes a mean teacher model to enhance training robustness with noisy labels to both stabilize the training accuracy curve for fitting in ACT and provide pseudo labels for correction in O2C. Moreover, O2C is implementedonlinewithout the need to store updated labels every training epoch. We validate our approach on two building footprint segmentation datasets with different spatial resolutions. Experimental results with varying degrees of building label noise demonstrate the robustness of AIO2. Source code will be available at https://github.com/zhu-xlab/AIO2.git.

Urban land use structures impact local climate conditions of metropolitan areas. To shed light on the mechanism of local climate wrt. urban land use, we present a novel, data-driven deep learning architecture and pipeline, DeepLCZChange, to correlate airborne LiDAR data statistics with the Landsat 8 satellite’s surface temperature product. A proof-of-concept numerical experiment utilizes corresponding remote sensing data for the city of New York to verify the cooling effect of urban forests.

Automatic label generation systems, which are capable to generate huge amounts of labels with limited human efforts, enjoy lots of potential in the deep learning era. These easy-to-come-by labels inevitably bear label noises due to a lack of human supervision and can bias model training to some inferior solutions. However, models can still learn some plausible features, before they start to overfit on noisy patterns. Inspired by this phenomenon, we propose a new Peaks fusion assisted EArly-Stopping (PEAS) approach for imagery segmentation with noisy labels, which is mainly composed of two parts. First, a fitting based early-stopping criterion is used to detect the turning phase from which models are about to mimic noise details. After that, a peaks fusion strategy is applied to select reliable models in the detection zone to generate final fusion results. Here, validation accuracies are utilized as indicators in model selection. The proposed method was evaluated on New York City dataset whose labels were automatically collected by a rule-based label generation system, thus noisy to some extent due to a lack of human supervision. The experimental results showed that the proposed PEAS method can achieve both promising statistical and visual results when trained with noisy labels.

Discovering ancient agricultural terraces in desert regions is important for the monitoring of long-term climate changes on the Earth’s surface. However, traditional ground surveys are both costly and limited in scale. With the increasing accessibility of aerial and satellite data, machine learning techniques bear large potential for the automatic detection and recognition of archaeological landscapes. In this paper, we propose a deep semantic model fusion method for ancient agricultural terrace detection. The input data includes aerial images and LiDAR generated terrain features in the Negev desert. Two deep semantic segmentation models, namely DeepLabv3+ and UNet, with EfficientNet backbone, are trained and fused to provide segmentation maps of ancient terraces and walls. The proposed method won the first prize in the International AI Archaeology Challenge. Codes are available at https://github.com/wangyi111/international-archaeologyai-challenge.

We present and evaluate a weakly-supervised methodology to quantify the spatiotemporal distribution of urban forests based on remotely sensed data with close-to-zero human interaction. Successfully training machine learning models for semantic segmentation typically depends on the availability of high-quality labels. We evaluate the benefit of high-resolution, three-dimensional point cloud data (LiDAR) as source of noisy labels in order to train models for the localization of trees in orthophotos. As proof of concept we sense Hurricane Sandy's impact on urban forests in Coney Island, New York City (NYC) and reference it to less impacted urban space in Brooklyn, NYC.

In this letter, we propose a new phase-induced Gabor-based multiview active learning (MVAL) (PGMVAL) approach for hyperspectral image (HSI) classification. Our main contribution is to explore the potential of the phase offset term$P$in hand-crafted Gabor feature extraction, which is rarely exploited in previous works. The Gabor filters with$P$added, named as the phase-induced Gabor filters, are able to adjust their frequency response characteristics through$P$. Specifically, we utilize the phase-induced Gabor filtering for view generation purposes under a MVAL framework. As a result, PGMVAL is capable to exploit the complementary information residing in the phase-induced Gabor features corresponding to different$P\text{s}$and simultaneously avoids high memory consumption and a large number of training samples required caused by introducing a new parameter. The experimental results obtained on two benchmark HSI data sets show that the proposed PGMVAL approach using phase-induced Gabor filtering could achieve better classification results with limited training samples.

In recent years, Gabor filtering has been successfully applied in spectral-spatial hyperspectral image (HSI) classification tasks due to its strong power to characterize surface materials. A standard Gabor filter involves the real and imaginary parts. A common way to jointly use both the two parts is in the form of Gabor magnitude features. However, this combination form might weaken some unique characteristics of the two parts, therefore leading to limited improvement when compared to the real parts only. To solve this problem, we propose an integrated Gabor-based decision fusion (IG-DF) method for HSI classification. As the term suggests, our approach explores the integration of the two parts of Gabor features by means of a decision-level fusion strategy, where an extensive number of Gabor cubes generated with the real and imaginary parts of Gabor filters as well as different frequencies and orientations are directly fed into each classifier to yield a set of probability outputs. Afterward, the integrated Gabor-based decision fusion strategy is adopted to generate the final classification results from these probability outputs. Our experimental results, conducted on two commonly-used HSI datasets, demonstrate that the proposed IG-DF approach exhibits good improvements when compared with the real-part based and the magnitude feature based methods.

Recently, many convolutional neural network (CNN) methods have been designed for hyperspectral image (HSI) classification since CNNs are able to produce good representations of data, which greatly benefits from a huge number of parameters. However, solving such a high-dimensional optimization problem often requires a large number of training samples in order to avoid overfitting. In addition, it is a typical nonconvex problem affected by many local minima and flat regions. To address these problems, in this article, we introduce the naive Gabor networks or Gabor-Nets that, for the first time in the literature, design and learn CNN kernels strictly in the form of Gabor filters, aiming to reduce the number of involved parameters and constrain the solution space and, hence, improve the performances of CNNs. Specifically, we develop an innovative phase-induced Gabor kernel, which is trickily designed to perform the Gabor feature learning via a linear combination of local low-frequency and high-frequency components of data controlled by the kernel phase. With the phase-induced Gabor kernel, the proposed Gabor-Nets gains the ability to automatically adapt to the local harmonic characteristics of the HSI data and, thus, yields more representative harmonic features. Also, this kernel can fulfill the traditional complex-valued Gabor filtering in a real-valued manner, hence making Gabor-Nets easily perform in a usual CNN thread. We evaluated our newly developed Gabor-Nets on three well-known HSIs, suggesting that our proposed Gabor-Nets can significantly improve the performance of CNNs, particularly with a small training set.

Spectral-spatial Gabor filtering, which is based on 3-D local harmonic analysis, has been a powerful spectral-spatial feature extraction tool for hyperspectral image (HSI) classification. However, existing spectral-spatial Gabor approaches are prone to oversmoothing, neglecting the existences of edges and negatively affecting the classification. In this article, we propose a new HSI Gabor filtering concept, called spectral-spatial-range Gabor filtering, which intends to restrain edge interference from disturbing local spectral-spatial harmonic components. Contributions and novelties of our work can be identified as follows: 1) an HSI filtering framework is created, which can accommodate various Gabor filtering procedures and hence offer the potential to guide the design of new Gabor filters; 2) following such a unified filtering framework and taking into consideration both local spectral-spatial harmonic characteristics and range domain variations, we develop a new concept of spectral-spatial-range Gabor filtering; and 3) utilizing this proposed Gabor prototype and elaborating mathematical derivations, we achieve a novel discriminative spectral-spatial-range Gabor filtering method, which can deal with discriminative local harmonics and edge interference simultaneously along the spectral-spatial-range domain, obtaining highly discriminative Gabor features while yielding linear computational complexity. Our novel method is evaluated on four real HSI data sets and achieves excellent performances.

In some remote sensing applications such as change detection, satellite images with both high spatial and high temporal resolution are required. However, no single satellite sensor can currently provide such images due to technical specifications. To solve this problem, spatio-temporal fusion provides a cost-effective solution. In this paper, we propose a new spatio-temporal fusion approach, based on convolutional neural networks (CNNs), for Landsat and MODIS image fusion. Specifically, the proposed approach utilizes CNNs to model the heterogeneity of fine pixels from the coarse MODIS images. Here, the heterogeneity of fine pixels is defined as the difference between the reflectance changes obtained from the two types of images. After that, two transition-predicted images can be obtained using the trained CNNs, which are then fused in order to obtain a fi-nal prediction. In our newly proposed approach, CNNs are only used to learn the heterogeneity of fine pixels rather than the whole images, thus providing a more stable and less time-consuming strategy as compared to other available approaches. We evaluated the proposed approach on a public spatio-temporal fusion dataset and the obtained results suggest that our newly developed method achieves state-of-the-art performance.

This work proposes a new collaborative active and semi-supervised learning approach, named accessibility-free active learning (AFAL), for hyperspectral imaging classification. The proposed approach aims to tackle an existing problem in traditional active learning methods, that is, the fact that some selected samples are not accessible by oracles for assigning them pseudo labels, i.e., confident predictions for the classifier. The proposal specifically addresses this problem using superpixels in a self-training context. Specifically, AFAL first generates a set of candidates locally around the labeled pixels and then expands them to other subregions via a density peak-based augmentation strategy, in order to guarantee the confidence of pseudo labels. Our experimental results, obtained on two real and well-used hyperspectral images, reveal that the proposed scheme can lead to state-of-the-art performance.

Imaging spectroscopy, also known as hyperspectral imaging, has been transformed in the last four decades from being a sparse research tool into a commodity product available to a broad user community. Specially, in the last 10 years, a large number of new techniques able to take into account the special properties of hyperspectral data have been introduced for hyperspectral data processing, where hyperspectral image classification, as one of the most active topics, has drawn massive attentions. Spectral-spatial hyperspectral image classification can achieve better classification performance than its pixel-wise counterpart, since the former utilizes not only the information of spectral signature but also that from spatial domain. In this paper, we provide a comprehensive overview on the methods belonging to the category of spectral-spatial classification in a relatively unified context. First, we develop a concept of spatial dependency system that involves pixel dependency and label dependency, with two main factors: neighborhood covering and neighborhood importance. In terms of the way that the neighborhood information is used, the spatial dependency systems can be classified into fixed, adaptive, and global systems, which can accommodate various kinds of existing spectral-spatial methods. Based on such, the categorizations of single-dependency, bilayer-dependency, and multiple-dependency systems are further introduced. Second, we categorize the performings of existing spectral-spatial methods into four paradigms according to the different fusion stages wherein spatial information takes effect, i.e., preprocessing-based, integrated, postprocessing-based, and hybrid classifications. Then, typical methodologies are outlined. Finally, several representative spectral-spatial classification methods are applied on real-world hyperspectral data in our experiments.

Active learning (AL) has obtained a great success in supervised remotely sensed hyperspectral image classification, since it is able to select highly informative training samples. As an intrinsically biased sampling approach, AL generally favors the selection of samples following discriminative distributions, which are located in low-density areas. However, hyperspectral data are often highly class-mixed, i.e., most samples fluctuate in the overlapping regions of distributions of different classes. In this case, the potential of AL to select effective training samples is more limited. As AL strongly depends on the features, a possibility to increase its capabilities is to transfer the data into a highly discriminative feature space, in which the mixture of distributions that different classes of data follow tends to reduce. Based on this observation, in this paper, we introduce the concept of feature-driven AL, namely, the sample selection is going to be conducted in a given optimized feature space whose superiority is measured by an overall error probability. For illustrative purposes, we used Gabor filtering and morphological profiles for instantiation. Our experimental results, obtained on three real hyperspectral data sets, indicate that the proposed approach can significantly improve the potential of AL for hyperspectral image classification.

Active learning has obtained a great success in supervised remotely sensed hyperspectral image classification, since it can be used to select highly informative training samples. As an intrinsically biased sampling approach, it generally favors the selection of samples following discriminative distributions, i.e., those located in low density areas in feature space. However, the hyperspectral data are often highly mixed, i.e., most samples fluctuate in a local density areas. In this case, the potential of active learning for effective training sample selection is more limited. In order to address this relevant issue, we develop a new Gabor-based active learning approach for hyperspectral image classification, which consists of two main steps. First, we use a Gabor filter for feature extraction, which aims at bringing the data into a discriminative space. Then, we perform active learning to find the most informative training samples in the low density areas prior to the final classification. Our experimental results, conducted using two real hyperspectral datasets, indicate that the proposed Gabor-based approach can greatly improve the potential of active learning for classification purposes.