While the pretraining of Foundation Models (FMs) for remote sensing (RS) imagery is on the rise, models remain restricted to a few hundred million parameters. Scaling models to billions of parameters has been shown to yield unprecedented benefits including emergent abilities, but requires data scaling and computing resources typically not available outside industry R&D labs. In this work, we pair high-performance computing resources including Frontier supercomputer, America's first exascale system, and high-resolution optical RS data to pretrain billion-scale FMs. Our study assesses performance of different pretrained variants of vision Transformers across image classification, semantic segmentation and object detection benchmarks, which highlight the importance of data scaling for effective model scaling. Moreover, we discuss construction of a novel TIU pretraining dataset, model initialization, with data and pretrained models intended for public release. By discussing technical challenges and details often lacking in the related literature, this work is intended to offer best practices to the geospatial community toward efficient training and benchmarking of larger FMs.

The design of a pretraining dataset is emerging as a critical component for the generality of foundation models. In the remote sensing realm, large volumes of imagery and benchmark datasets exist that can be leveraged to pretrain foundation models, however using this imagery in absence of a well-crafted sampling strategy is inefficient and has the potential to create biased and less generalizable models. Here, we provide a discussion and vision for the curation and assessment of pretraining datasets for remote sensing geospatial foundation models. We highlight the importance of geographic, temporal, and image acquisition diversity and review possible strategies to enable such diversity at global scale. In addition to these characteristics, support for various spatial-temporal pretext tasks within the dataset is also critical. Ultimately, our primary objective is to place emphasis on and draw attention to the data curation stage of the foundation model development pipeline. By doing so, we think it is possible to reduce biases of geospatial foundation models, as well as enable broader generalization to downstream remote sensing tasks and applications.

Population growth is increasingly happening in slum settlements of the large urban centers in the Global South. The term "slum" encompasses a wide range of communities, located mostly in underserved areas, and often exhibiting distinct structural and functional informalities with a relatively high concentration of marginalized populations. To address the issues confronting slums for effective planning and development, including the realistic estimation of the resident population, identifying them accurately is fundamental. Given the disagreements over a universal definition, diverse characteristic features, and socio-political limitations, global detection of slums is a veritable challenge. In this paper, we present experiments in slum detection using a scene classification algorithm and 3-meter spatial resolution satellite imagery. We train and evaluate the model for slum detection in Mumbai, India for the year 2023 and test the temporal generalization of the trained model on Mumbai in 2020 and 2018. In addition, we explore the pathways toward geographic generalization to Kolkata and Delhi (India). We discuss several limitations in the workflow and model, situate our findings in the existing literature, and suggest improvements and alternatives. With this, we establish baseline methods and experiments as a first step towards developing an image-based global slum detection framework and algorithm. This work adds to the community discussion on methods, data challenges, and open questions related to the detection of slums globally. With this research, we hope to improve our understanding of human settlements, especially in critical areas, improve population estimates, and help measure progress towards the sustainable development goals.

Rapid building damage assessment (BDA) is vital in guiding disaster response missions and estimating population distribution across impacted areas. While commercial satellite imagery providers have enabled near-daily monitoring of the Earth, near-realtime assessment of disaster scenarios frequently requires analysis of off-nadir imagery, as satellites are often far from impacted areas for at-nadir post-event imaging to occur Such scenarios are, however, underrepresented in existing BDA datasets and methodologies. With this motivation, we investigate generalization capabilities of current BDA practices across overhead view-angles and strategies for their improvement. Using a labeled dataset of images capturing conflict-related damages, we first train a baseline BDA architecture using imbalanced and balanced datasets with respect to view-angle. Then, we explore conditional convolutions parameterized on image features, image nadir, and their combination as a mechanism for conditioning on view-angles. Experiments demonstrate the limitations of current practice and the potential of conditional mechanisms to increase model robustness to view-angle variations.

Computer vision algorithms are increasingly leveraged to accelerate geospatial analysis for disaster response and recovery. As the diversity of remote sensing imagery grows with optical, SAR, and other modalities, a perquisite for analytics is cross-modal image registration. There is a high potential to harness computer vision for this pre-processing requirement toward enabling downstream analytics such as heterogeneous change detection, automated feature extraction, and data fusion. Advancement in these areas has the potential to simplify data wrangling tasks and further accelerate disaster response timelines. The SpaceNet 9 challenge (launching in mid-2024) focuses on addressing the cross-modal image registration problem and demonstrating the utility of such modules on earthquake impacted scenarios. This paper describes the motivation for the SpaceNet 9 and provides a first overview of the dataset, the baseline algorithm, and implications for seeking cross-modal image registration in Earth observation. Code is available at https://github.com/SpaceNetChallenge/SpaceNet9.

In recent years, deep learning has become an increasingly popular alternative for modeling in geoscience applications due to its scalability and efficiency. However, the interpretability, compute, data volume, and hyperparameter tuning requirements of deep learning models make development and monitoring difficult. Furthermore, model explainability and communicating results obtained by these models to users or domain experts is a challenge, as domain experts in geoscience also need to have a deep understanding of how those models function in order to support their scientific works. Here, we describe a science gateway and machine learning pipeline for predicting gravity anomalies from geophysical data. The gateway, built on open-source technologies, provides a holistic view of the pipeline through interactive visualizations aimed at enabling efficient exploratory data analysis. The repeatability, reproducibility, and monitoring capabilities of this overall system allow us to iterate and analyze at scale. Using this pipeline and gateway, we can repeatedly produce accurate high-resolution gravity anomaly datasets. By describing the underlying technologies, implementation, and results, we provide a foundation for the broader adoption of science gateways into cross-cutting geoscience and machine learning research projects as a means to improve the scientific discovery and collaboration in the geophysics and computational sciences community.

The development of algorithms to assess the effects of natural disasters plays an integral role in response efforts. There is a growing opportunity to leverage remote sensing data and computer vision to quickly analyze the scale of damage and organize a humanitarian response when extreme weather events occur. By automating the process of identifying damage to roads and infrastructure, we can significantly reduce response time, directing relief efforts on a time scale of minutes or hours rather than days. The SpaceNet 8 challenge featured a complex multi-class segmentation problem in the context of flood detection from remote sensing imagery. Competitors were tasked with leveraging both pre- and post-flooding event imagery to detect buildings and roads, as well as identify which of these object instances were affected by the flooding event. We examine the outcome of the SpaceNet 8 challenge and present an overview of the competition and a deeper look at the top-performing submissions.

Given the tremendous volume of accessible Earth Observation (EO) data, there is a need to develop scalable Geospatial Artificial Intelligence (GeoAI) solutions for time-sensitive applications. Scalability in this context refers to rapidly processing large-scale EO data using high performance computing resources. Accurate mapping of the built environment from remote sensing (RS) imagery has been one of the crucial components in GeoAI workflows for a wide spectrum of humanitarian applications. Derived vector data of built environment is often leveraged for disaster preparedness and response activities. However, factors such as differences in ortho-rectification, atmospheric conditions and human error, results in spatial misalignment between vector data and the timely available RS imagery. Model training for downstream tasks such as object detection, change analysis, etc., is negatively impacted due to such spatial misalignment. Although there has been progress towards automatic alignment of vector data, the lack of scalability remains an open research challenge. This paper proposes to leverage parallel computing to optimize an automatic vector data alignment workflow. It further employs CPU-level multi-core parallelism for improving the performance of the workflow for scalable built environment mapping. We report observations and discuss findings from the preliminary experiments performed on the Summit Supercomputer.

Deep learning has proven its effectiveness in numerous tasks for remote sensing scene understanding. However there is an increasing interest to explore fusion of domain-specific background information to the deep neural network to further improve its performance. Remote sensing researchers are also working towards developing models that generalize and adapt to multiple applications. Generalization challenges coupled with the scarcity of large corpora of high-quality noise-free labelled data, have together fueled an interest for leveraging background information. Knowledge graphs serve as excellent choice to represent domain-specific information in a structured, standardized and extensible manner. Integrating symbolic knowledge representations in the form of Knowledge Graph Embedding (KGE) to perform neuro-symbolic reasoning is an emerging research direction promising significant impacts. This vision paper seeks to position ideas and provoke early thoughts toward advancing neuro-symbolic artificial intelligence in the context of geospatial challenges. Specifically, it conceptualizes and elaborates on an architecture for infusing geospatial knowledge from knowledge graph in a deep neural network pipeline. As guiding case studies - land-use land-cover classification, object detection and instance segmentation can benefit from infusing spatio-contextual information with remote sensing imagery. The discussion further reflects on and articulates the challenges and explainable AI opportunities anticipated when scaling and maintaining large-scale geospatial knowledge graphs.

Understanding population at risks has been a focus of the LandScan program through its development of population estimates. With advancements in computer vision, deep learning technologies and access to High Performance Computing (HPC) and high resolution imagery, population estimates are now modeled at the building level. However, when those patterns are disrupted, rapid updates to population distribution estimates are needed to support humanitarian aid and response. Oak Ridge National Laboratory (ORNL) recently adapted an existing deep learning building footprint extraction model in development of a scalable approach to Building Damage Assessments (BDA). This new opportunity opens the possibility of automating BDA to support rapid population distribution estimate updates for geographic areas involved in geopolitical conflicts or natural events for humanitarian aid and response or where to focus recovery efforts. In addition, incorporate social surveys to further model human behavior under conflict or other scenarios that disrupt normal patterns of life.

Floods are one of the major types of natural disasters responsible for loss of life, destruction of buildings and infrastructure, erosion of arable land, and environmental hazards around the world. Climate change, increasing populations, and urbanisation of flood plains will only increase the risk of flooding in the next few years. SpaceNet 8 presents a dataset that combines building footprint detection, road network extraction, and flood detection covering 850km2, including ~32,000 buildings and ~ 1,300 km of roads, of which ~ 13% and ~ 15% are flooded, respectively.

Model generalization remains a key challenge in the analysis of large amounts of heterogeneous satellite image data. One major limiting factor in developing generalizable models, in the context of supervised learning, is the lack of high quality training datasets. A model's capacity to perform well on new data is often inhibited by imbalance and bias in the data that was used for training. This is especially a problem when using convolutional neural networks to classify urban land-use in satellite images. Notable dataset imbalance issues in this application include land-use type imbalance and image scene imbalance. To begin understanding these dataset imbalance problems in more detail, we develop and test a number of sampling methods for generating training image datasets from subjective training polygons for urban land-use classification. We investigate sampling at different point densities as a means to reduce content repetition and therefore content imbalance and bias in the training image dataset.

Machine learning and deep learning provide a means for generating urban land-use maps with relatively little human effort compared to manually digitizing images. This is especially important for supporting global and regional initiatives focused on sustainability, planning, health, pro-poor policy, infrastructure, and population distribution estimates. Many of these initiatives work in areas where geospatial data is scarce, such as the global south, and often use land-use maps to help achieve their goals. In this study, we develop a typology for automated labeling of urban land-use data that captures the variation in structural patterns within cities. A comparison of classification accuracy between convolutional neural networks (CNNs) and support vector machines (SVMs) coupled with handcrafted features is conducted. Through experimental validation on two highly dense cities in Africa, we report on new insights and the potential benefits offered by both multiscale handcrafted features and multiscale-CNNs even with limited training data.