While the pretraining of Foundation Models (FMs) for remote sensing (RS) imagery is on the rise, models remain restricted to a few hundred million parameters. Scaling models to billions of parameters has been shown to yield unprecedented benefits including emergent abilities, but requires data scaling and computing resources typically not available outside industry R&D labs. In this work, we pair high-performance computing resources including Frontier supercomputer, America's first exascale system, and high-resolution optical RS data to pretrain billion-scale FMs. Our study assesses performance of different pretrained variants of vision Transformers across image classification, semantic segmentation and object detection benchmarks, which highlight the importance of data scaling for effective model scaling. Moreover, we discuss construction of a novel TIU pretraining dataset, model initialization, with data and pretrained models intended for public release. By discussing technical challenges and details often lacking in the related literature, this work is intended to offer best practices to the geospatial community toward efficient training and benchmarking of larger FMs.

The design of a pretraining dataset is emerging as a critical component for the generality of foundation models. In the remote sensing realm, large volumes of imagery and benchmark datasets exist that can be leveraged to pretrain foundation models, however using this imagery in absence of a well-crafted sampling strategy is inefficient and has the potential to create biased and less generalizable models. Here, we provide a discussion and vision for the curation and assessment of pretraining datasets for remote sensing geospatial foundation models. We highlight the importance of geographic, temporal, and image acquisition diversity and review possible strategies to enable such diversity at global scale. In addition to these characteristics, support for various spatial-temporal pretext tasks within the dataset is also critical. Ultimately, our primary objective is to place emphasis on and draw attention to the data curation stage of the foundation model development pipeline. By doing so, we think it is possible to reduce biases of geospatial foundation models, as well as enable broader generalization to downstream remote sensing tasks and applications.

Computer vision algorithms are increasingly leveraged to accelerate geospatial analysis for disaster response and recovery. As the diversity of remote sensing imagery grows with optical, SAR, and other modalities, a perquisite for analytics is cross-modal image registration. There is a high potential to harness computer vision for this pre-processing requirement toward enabling downstream analytics such as heterogeneous change detection, automated feature extraction, and data fusion. Advancement in these areas has the potential to simplify data wrangling tasks and further accelerate disaster response timelines. The SpaceNet 9 challenge (launching in mid-2024) focuses on addressing the cross-modal image registration problem and demonstrating the utility of such modules on earthquake impacted scenarios. This paper describes the motivation for the SpaceNet 9 and provides a first overview of the dataset, the baseline algorithm, and implications for seeking cross-modal image registration in Earth observation. Code is available at https://github.com/SpaceNetChallenge/SpaceNet9.

Machine learning models have been extensively used for analyzing Earth Observation images and have played a crucial role in advancing the field. While most studies focus on improving the model’s performance, some aim to understand the model’s output. These explainable approaches provide reasoning behind the model’s output, establishing trust and confidence in the results. However, the evaluation of these models’ performance is mainly based on accuracy. To enhance the fairness and transparency of machine learning models, the evaluation of these models on Earth Observation images should also focus on explainability. This work reflects on existing research on explaninable AI in Remote Sensing and further outlines the desirable properties of the gold standard metric for evaluating explainable machine learning models on EO images.

Archives of remote sensing (RS) data are increasing swiftly as new sensing modalities with enhanced spatiotemporal resolution become operational. While promising new breakthroughs, the sheer volume of RS archives stretches the limits of human analysts and existing AI tools, as most models are: i) limited to single data modalities; ii) task-specific; iii) heavily reliant on labeled data. The emerging Foundation Models (FMs) have the potential to address these limitations. Trained on vast unlabeled datasets through self-supervised learning, FMs enable generic feature extraction that facilitate specialization to a wide variety of downstream tasks. This paper describes a vision towards an FM for multimodal Earth Observation data (FM4EO), discussing key building blocks and open challenges. We put particular emphasis on multimodal reasoning, a topic underexplored in EO. Our ultimate goal is a practical path toward FM4EO with capacity to unlock breakthroughs in few-shot learning scenarios, multimodal geographic knowledge integration, synthesis, and hypothesis generation.

Given the tremendous volume of accessible Earth Observation (EO) data, there is a need to develop scalable Geospatial Artificial Intelligence (GeoAI) solutions for time-sensitive applications. Scalability in this context refers to rapidly processing large-scale EO data using high performance computing resources. Accurate mapping of the built environment from remote sensing (RS) imagery has been one of the crucial components in GeoAI workflows for a wide spectrum of humanitarian applications. Derived vector data of built environment is often leveraged for disaster preparedness and response activities. However, factors such as differences in ortho-rectification, atmospheric conditions and human error, results in spatial misalignment between vector data and the timely available RS imagery. Model training for downstream tasks such as object detection, change analysis, etc., is negatively impacted due to such spatial misalignment. Although there has been progress towards automatic alignment of vector data, the lack of scalability remains an open research challenge. This paper proposes to leverage parallel computing to optimize an automatic vector data alignment workflow. It further employs CPU-level multi-core parallelism for improving the performance of the workflow for scalable built environment mapping. We report observations and discuss findings from the preliminary experiments performed on the Summit Supercomputer.

Deep learning has proven its effectiveness in numerous tasks for remote sensing scene understanding. However there is an increasing interest to explore fusion of domain-specific background information to the deep neural network to further improve its performance. Remote sensing researchers are also working towards developing models that generalize and adapt to multiple applications. Generalization challenges coupled with the scarcity of large corpora of high-quality noise-free labelled data, have together fueled an interest for leveraging background information. Knowledge graphs serve as excellent choice to represent domain-specific information in a structured, standardized and extensible manner. Integrating symbolic knowledge representations in the form of Knowledge Graph Embedding (KGE) to perform neuro-symbolic reasoning is an emerging research direction promising significant impacts. This vision paper seeks to position ideas and provoke early thoughts toward advancing neuro-symbolic artificial intelligence in the context of geospatial challenges. Specifically, it conceptualizes and elaborates on an architecture for infusing geospatial knowledge from knowledge graph in a deep neural network pipeline. As guiding case studies - land-use land-cover classification, object detection and instance segmentation can benefit from infusing spatio-contextual information with remote sensing imagery. The discussion further reflects on and articulates the challenges and explainable AI opportunities anticipated when scaling and maintaining large-scale geospatial knowledge graphs.

Understanding population at risks has been a focus of the LandScan program through its development of population estimates. With advancements in computer vision, deep learning technologies and access to High Performance Computing (HPC) and high resolution imagery, population estimates are now modeled at the building level. However, when those patterns are disrupted, rapid updates to population distribution estimates are needed to support humanitarian aid and response. Oak Ridge National Laboratory (ORNL) recently adapted an existing deep learning building footprint extraction model in development of a scalable approach to Building Damage Assessments (BDA). This new opportunity opens the possibility of automating BDA to support rapid population distribution estimate updates for geographic areas involved in geopolitical conflicts or natural events for humanitarian aid and response or where to focus recovery efforts. In addition, incorporate social surveys to further model human behavior under conflict or other scenarios that disrupt normal patterns of life.

There has been an increase in the adoption of Linked Data and subsequently representing data in the form of knowledge graphs across a wide spectrum of domains. There has also been significant interest in the remote sensing community to publish Earth Observation data in the form of Linked Data. As the geospatial Linked Data cloud on the internet grows, there arises a need for efficient methods of exploratory analysis of such information-rich geospatial knowledge graphs. Knowledge graph representation of remote sensing scenes has proved to add significant value for effective mining of implicit information in addition to seamless integration with other data sources. This work is geared towards visual exploration of semantically enriched Remote Sensing Scene Knowledge Graphs (RSS-KGs). In this paper, we propose and implement an interactive web-based interface to visually explore and interact with RSS-KGs using Cesium. The proposed interface seeks to visualize the know ledge graph in the form of nodes and edges, mapped over the remote sensing scene consisting of different land use land cover regions and their inferred characteristics in addition to their spatial relationships with one another. It is envisaged that visualization in the form of nodes and edges would aid in visually validating the spatial relations in the knowledge graph, thus enhancing the understanding of the geospatial knowledge graph from the end user perspective. We demonstrate the efficacy of the interface through the visual exploration of an enriched geospatial knowledge graph of a remote sensing scene captured during an urban flood event.

Remote sensing lidar point cloud dataset embeds inherent 3D topological, topographical and complex geometrical information which possess immense potential in applications involving machine-understandable 3D perception. The lidar point clouds are unstructured, unlike images, and hence are challenging to process. In our work, we are exploring the possibility of deep learning-based question-answering on the lidar 3D point clouds. We are proposing a deep CNN-RNN parallel architecture to learn lidar point cloud features and word embedding from the questions and fuse them to form a feature mapping for generating answers. We have restricted our experiments for the urban domain and present preliminary results of binary question-answering (yes/no) using the urban lidar point clouds based on the perplexity, edit distance, evaluation loss, and sequence accuracy as the performance metrics. Our proposed hypothesis of lidar question-answering is the first attempt, to the best of our knowledge, and we envisage that our novel work could be a foundation in using lidar point clouds for enhanced 3D perception in an urban environment. We envisage that our proposed lidar question-answering could be extended for machine comprehension-based applications such as rendering lidar scene descriptions and content-based 3D scene retrieval.

In the early hours following the earthquake, supporting humanitarian actions like rescue operations and relief distribution is the primary objective of the rescue managers. The damage mapping can be performed using reliable data that can be obtained from high-resolution satellite imagery but obtaining satellite imagery can be challenging for some days post disaster due to revisit time. Considering the disaster response timing, Unmanned Aerial Vehicles (UAV) are used because ground transportation systems are ineffective due to road blockage. In this work, we make use of Light Detection and Ranging (LiDAR) 3D point cloud data obtained for Haiti Earthquake. The focus of our work is to develop and implement an approach for LiDAR data classification to enable Earthquake damage mapping and detection. This is obtained by running our deep learning network on NVIDIA Jetson Nano embedded supercomputing platform. This approach takes the advantage of embedded High-Performance computing and low power consumption capabilities of Jetson Nano which enhances the classification and promotes rapid response which is the key to manage post-disaster activities. Jetson Nano is a feasible option which provides a GPU architecture that is optimized for running energy-aware deep learning models and which generates the results in real or near-real time. We envisage that our work could be extended to perform near real-time classification of LiDAR point clouds in a post earthquake scenario.

With an increase in Geospatial Linked Open Data being adopted and published over the web, there is a need to develop intuitive interfaces and systems for seamless and efficient exploratory analysis of such rich heterogeneous multi-modal datasets. This work is geared towards improving the exploration process of Earth Observation (EO) Linked Data by developing a natural language interface to facilitate querying. Questions asked over Earth Observation Linked Data have an inherent spatio-temporal dimension and can be represented using GeoSPArql. This paper seeks to study and analyze the use of RNN-based neural machine translation with attention for transforming natural language questions into GeoSPArql queries. Specifically, it aims to assess the feasibility of a neural approach for identifying and mapping spatial predicates in natural language to GeoSPARQL's topology vocabulary extension including - Egenhofer and RCC8 relations. The queries can then be executed over a triple store to yield answers for the natural language questions. A dataset consisting of mappings from natural language questions to GeoSPArql queries over the Corine Land Cover(CLC) Linked Data has been created to train and validate the deep neural network. From our experiments, it is evident that neural machine translation with attention is a promising approach for the task of translating spatial predicates in natural language questions to GeoSPArql queries.

Real-time embedded vision tasks require extraction of complex geometric and morphological features from the raw 3D point cloud acquired using range scanning systems like lidar, radar etc. and depth cameras. Such applications are found in autonomous navigation, surveying, 3D mapping and localization tasks such as automatic target recognition (ATR). Typically, a dataset acquired during surveying by remote sensing lidar scanners, known as point cloud, is (1) huge in size and requires a big chunk of memory for processing at a single instance and, (2) experiences missing information due to rapid change in orientation of the sensor while scanning. In our work, we are addressing both the issues combinedly by proposing an online point cloud super-resolution approach for translating a low dimensional point cloud to a high dimensional dense point cloud by learning dictionaries in the low-dimensional subspace. We are presenting our approach for an urban road scenario by reconstructing dense point clouds of 3D objects and comparing results based on PSNR and Hausdorff distance.

Very High Resolution (VHR) remote sensing optical imagery is a huge source of information that can be utilized for earthquake damage detection and assessment. Time critical task such as performing the damage assessment, providing immediate delivery of relief assistance require immediate response; however, processing voluminous VHR imagery using highly accurate, but computationally expensive deep learning algorithms demands the High Performance Computing (HPC) power. To maximize the accuracy, deep convolution neural network (CNN) model is designed especially for the earthquake damage detection using remote sensing data and implemented using high performance GPU without compromising with the execution time. Geoeye1 VHR disaster images of the Haiti earthquake occurred in year 2010 is used for analysis. Proposed model provides good accuracy for damage detection; also significant execution speed is observed on GPU K80 High Performance Computing (HPC) platform.

In disasters, since time is of the essence, quick decision making based on actionable insights is desired. In our earlier work, we have demonstrated that the spatial relationships-based queries can play a vital role in the disaster response phase. However, we found that the utilization of spatial relationships rules (i.e. encoded spatial knowledge) via rule reasoning process do not scale well with the increased number of image regions. Most of the available Resource Description Framework (RDF) triplestores do not support rule reasoning due to the computational complexity and undecidable nature of the rule reasoning process. In this paper, we propose an alternative approach for utilizing spatial knowledge encoded in the form of spatial relationship rules. The proposed approach reformulates the spatial query by expanding it with the configuration encoded in the corresponding spatial relationship rule. The preliminary results are promising and show the applicability of the proposed approach during the time critical events such as flood disaster.

Natural disasters such as floods, earthquakes, hurricanes, etc. have a huge impact on a society-causing destruction of life and property in their wake. During disasters such as flood, it is crucial to understand the dynamics of the situation as it occurs for effective response. In this paper, we address the problem of satellite image classification for urban floods using deep learning. We propose an encoder-decoder neural network based on the Efficient Residual Factorized Convnet(ERFNet), for multi-class segmentation of urban floods from multi-spectral satellite imagery. The ERFNet architecture capitalizes on skip connections and one dimensional convolutions to achieve the best possible trade-off between accuracy and efficiency. Since time is of essence during a disaster, the choice of the ERFNet architecture on a high performance computing (HPC) platform is apt. Satellite imagery from WorldView-2 of floods in Srinagar, India during September 2014 have been used for this study. The tool `markGT' has been developed to assist end-to-end annotation of satellite imagery. The urban flood dataset used for this study has been generated using markGT. The proposed deep learning model over urban flood satellite imagery gives promising results on Nvidia Tesla K80 GPU. We envisage that the proposed model could be extended and improved for real-time classification of urban floods, thereby aiding disaster response personnel in making informed decisions.

Disasters such as earthquakes, floods, landslides etc. create great economic and social loss by destroying the balance of life and property and create chaos. In the wake of a disaster, it becomes very significant to take real-time and on-the-fly actions to minimize the effects of the event. Remote Sensing data acquired through airborne or spaceborne platforms is usually huge in size and requires huge time in generating actionable insights during the disaster scenario. In this work, we propose a two-fold analysis of the Very High Resolution (VHR) satellite imagery based on Compressive Sensing (CS) and Deep Learning. We propose employing a deep learning approach for inferencing over compressed sensing satellite imagery. We hypothesize that this could be beneficial in generating real-time actionable insights during a catastrophe. In our work, we are using the satellite imagery from GeoEye-1 of Haiti Earthquake. Our objectives are: (1) To generate CS images for 75%, 50%, and, 25% sampling on the sparse space and (2) To develop a deep learning pixel-level classification model based on the UNet architecture using the original and reconstructed images. The UNet architecture has shown promising results for pixel-level classification in the recent literature. We envisage to combine both the objectives into an end-to-end learning framework for on-board processing which we foresee would be of great significance in various applications for rapid disaster management response.

Numerous remote sensing applications - flood monitoring, forest fires monitoring, earthquake analysis etc. require users to query satellite images based on their content. Such requirements have led to the evolution of Content-based Image Information Mining Systems over the last decade. Recent developments in the area of Image Information Mining(IIM) are geared towards bridging the gap between low level image features and higher-level semantics. This research focuses on improving the semantic understanding of a remote sensing scene during the flood disaster from a spatio-contextual standpoint. During a flood occurrence, it is crucial to understand the flood inundation and receding patterns in context to the spatial configurations of the land-use/land-cover in the flooded regions. This study focuses on bridging the spatio-contextual semantic gap in understanding of the remote sensing imagery during a flood, thereby attempting to improve the machine interpretability of a flood remote sensing imagery. In this regard, the Flood Scene Ontology (FSO) has been developed to mine the topological and directional knowledge in context to the flood disaster phenomenon. The FSO is envisaged to form the basis for developing applications that would utilize the spatio-contextual semantics of the flood disaster to aid in the Disaster Assessment and Management process. This paper describes the conceptual framework that was developed to address the same.

Jetson TK1 is the first mobile processor from NVIDIA having similar features and architecture as that of a modern desktop GPU and still using low power from a mobile chip. Therefore, Jetson TK1 runs the same CUDA code (running on desktop GPU) with similar level of performance. Also, with the dawn of GPU technology, it has become possible to perform tasks (that are computationally intensive) in realtime or near-real time. In the agricultural domain, retrieving the biophysical parameters of the crop is important as it provides insights into the plant growth status. Inversion of the Radiative Transfer Model enables to obtain these parameters. However, such a process is highly computationally intensive. The focus of this work is to develop and implement an approach that takes the advantage of embedded High-Performance Computing (HPC) capability of Jetson TK1 to significantly improve the inversion process of a Radiative Transfer Model. The experimental results show that Jetson TK1 based biophysical parameters estimation gives significant speedup, which opens-up the possibility of having a Jetson based embedded platform for on-board biophysical parameters estimation in the future. In such a scenario, where there are constraints related to energy and power, Jetson TK1 can become a practicable option by providing a GPU based architecture for running energy-aware computationally intensive algorithms in parallel for processing the data, and generating the results in real-time or near-real time while taking care of the power usage.

During an extreme event such as flood disaster, it is important to study flood inundation and receding patterns to understand the dynamic spatio-temporal behavior of flood. In addition to the general change detection techniques in RS, a proper conceptualization of `change' during flood disaster is necessary to model its dynamic behavior. This motivated the development of Ontology, which is able to capture the dynamically evolving phenomenon. This Ontology is envisaged as a precursor for developing applications that integrate the spatio-temporal dimensions of a dynamically evolving system such as floods. This paper describes the conceptual framework that was developed to address the same.