Deployment of neural networks on resource-constrained devices demands models that are both compact and robust to adversarial inputs. However, compression and adversarial robustness often conflict. In this work, we introduce a dynamical low-rank training scheme enhanced with a novel spectral regularizer that controls the condition number of the low-rank core in each layer. This approach mitigates the sensitivity of compressed models to adversarial perturbations without sacrificing clean accuracy. The method is model- and data-agnostic, computationally efficient, and supports rank adaptivity to automatically compress the network at hand. Extensive experiments across standard architectures, datasets, and adversarial attacks show the regularized networks can achieve over 94 compression while recovering or improving adversarial accuracy relative to uncompressed baselines.

While the pretraining of Foundation Models (FMs) for remote sensing (RS) imagery is on the rise, models remain restricted to a few hundred million parameters. Scaling models to billions of parameters has been shown to yield unprecedented benefits including emergent abilities, but requires data scaling and computing resources typically not available outside industry R&D labs. In this work, we pair high-performance computing resources including Frontier supercomputer, America's first exascale system, and high-resolution optical RS data to pretrain billion-scale FMs. Our study assesses performance of different pretrained variants of vision Transformers across image classification, semantic segmentation and object detection benchmarks, which highlight the importance of data scaling for effective model scaling. Moreover, we discuss construction of a novel TIU pretraining dataset, model initialization, with data and pretrained models intended for public release. By discussing technical challenges and details often lacking in the related literature, this work is intended to offer best practices to the geospatial community toward efficient training and benchmarking of larger FMs.

In this paper, we address the need for extracting two key building elevation attributes—Lowest Adjacent Grade (LAG) and Highest Adjacent Grade (HAG)—which are crucial for effective flood risk management. Conventional methods, involving onsite surveying or the use of optical imagery-derived building footprints combined with Digital Elevation Models (DEMs), often face misalignment and time discrepancy issues due to varied remote sensing sources. We introduce a new, scalable method that exclusively relies on airborne LiDAR data to overcome these challenges. Our approach employs an object-based ground filtering technique, and the results were evaluated using two different DEMs and building footprint sets. The findings demonstrate that our single-source method, utilizing only airborne LiDAR data, significantly improves the accuracy of LAG and HAG calculations compared to traditional methods that use hand-digitized building footprints. The proposed approach offers a solution for comprehensive flood risk management endeavors.

In recent years, deep learning has become an increasingly popular alternative for modeling in geoscience applications due to its scalability and efficiency. However, the interpretability, compute, data volume, and hyperparameter tuning requirements of deep learning models make development and monitoring difficult. Furthermore, model explainability and communicating results obtained by these models to users or domain experts is a challenge, as domain experts in geoscience also need to have a deep understanding of how those models function in order to support their scientific works. Here, we describe a science gateway and machine learning pipeline for predicting gravity anomalies from geophysical data. The gateway, built on open-source technologies, provides a holistic view of the pipeline through interactive visualizations aimed at enabling efficient exploratory data analysis. The repeatability, reproducibility, and monitoring capabilities of this overall system allow us to iterate and analyze at scale. Using this pipeline and gateway, we can repeatedly produce accurate high-resolution gravity anomaly datasets. By describing the underlying technologies, implementation, and results, we provide a foundation for the broader adoption of science gateways into cross-cutting geoscience and machine learning research projects as a means to improve the scientific discovery and collaboration in the geophysics and computational sciences community.

Archives of remote sensing (RS) data are increasing swiftly as new sensing modalities with enhanced spatiotemporal resolution become operational. While promising new breakthroughs, the sheer volume of RS archives stretches the limits of human analysts and existing AI tools, as most models are: i) limited to single data modalities; ii) task-specific; iii) heavily reliant on labeled data. The emerging Foundation Models (FMs) have the potential to address these limitations. Trained on vast unlabeled datasets through self-supervised learning, FMs enable generic feature extraction that facilitate specialization to a wide variety of downstream tasks. This paper describes a vision towards an FM for multimodal Earth Observation data (FM4EO), discussing key building blocks and open challenges. We put particular emphasis on multimodal reasoning, a topic underexplored in EO. Our ultimate goal is a practical path toward FM4EO with capacity to unlock breakthroughs in few-shot learning scenarios, multimodal geographic knowledge integration, synthesis, and hypothesis generation.

Given the tremendous volume of accessible Earth Observation (EO) data, there is a need to develop scalable Geospatial Artificial Intelligence (GeoAI) solutions for time-sensitive applications. Scalability in this context refers to rapidly processing large-scale EO data using high performance computing resources. Accurate mapping of the built environment from remote sensing (RS) imagery has been one of the crucial components in GeoAI workflows for a wide spectrum of humanitarian applications. Derived vector data of built environment is often leveraged for disaster preparedness and response activities. However, factors such as differences in ortho-rectification, atmospheric conditions and human error, results in spatial misalignment between vector data and the timely available RS imagery. Model training for downstream tasks such as object detection, change analysis, etc., is negatively impacted due to such spatial misalignment. Although there has been progress towards automatic alignment of vector data, the lack of scalability remains an open research challenge. This paper proposes to leverage parallel computing to optimize an automatic vector data alignment workflow. It further employs CPU-level multi-core parallelism for improving the performance of the workflow for scalable built environment mapping. We report observations and discuss findings from the preliminary experiments performed on the Summit Supercomputer.

Deep learning has proven its effectiveness in numerous tasks for remote sensing scene understanding. However there is an increasing interest to explore fusion of domain-specific background information to the deep neural network to further improve its performance. Remote sensing researchers are also working towards developing models that generalize and adapt to multiple applications. Generalization challenges coupled with the scarcity of large corpora of high-quality noise-free labelled data, have together fueled an interest for leveraging background information. Knowledge graphs serve as excellent choice to represent domain-specific information in a structured, standardized and extensible manner. Integrating symbolic knowledge representations in the form of Knowledge Graph Embedding (KGE) to perform neuro-symbolic reasoning is an emerging research direction promising significant impacts. This vision paper seeks to position ideas and provoke early thoughts toward advancing neuro-symbolic artificial intelligence in the context of geospatial challenges. Specifically, it conceptualizes and elaborates on an architecture for infusing geospatial knowledge from knowledge graph in a deep neural network pipeline. As guiding case studies - land-use land-cover classification, object detection and instance segmentation can benefit from infusing spatio-contextual information with remote sensing imagery. The discussion further reflects on and articulates the challenges and explainable AI opportunities anticipated when scaling and maintaining large-scale geospatial knowledge graphs.

Disaster response requires timely damage assessment to prioritize rescue and restoration resources. However, providing critical and actionable knowledge after a natural disaster can be challenging due to the scale and the type of damages. This paper describes how remote sensing and machine learning techniques can be used to support rapid structure detection in the wake of a disaster. We use high resolution satellite imagery to identify structures on Hawaii's Big Island to support the Federal Emergency Management Agency's response efforts during the 2018 Kİlauea lava flow incident. This framework specifically showcases the generalizability of CNN models with no need to collect additional training samples to quickly map structures in pre- and post-event imagery and provide timely information to assist government agencies evaluating the extent and potential loss of disaster. With this case study, we further point out future directions to benefit similar larger scale efforts based on the lessons learned.

Deep learning based change detection methods have achieved the state-of-the-art performance in several recent studies. However, such methods usually are supervised, and therefore a large number of training samples is often a requisite. Manually preparing those training samples is not only expensive but also time-consuming, which does not fit the need of rapidly mapping the impacted area caused by nature disaster for further rescue mission and damage assessment. In this study, a fully automatic method was proposed to address the issue by automating training sample generation for mapping the impacted area caused by nature disaster. We used the 2011 tornado event in Joplin, Missouri, US, as an example of its application. The generated impacted area map was both visually and quantitatively evaluated against the ground truth data collected by US Federal Emergency Management Agency (FEMA). The results show that the map matches well with the FEMA ground truth data with 86% of major-damaged and destroyed buildings identified by FEMA on the ground also detected by this fully automatic framework using very high resolution (VHR) satellite images.

Supervised semantic segmentation methods provide state-of-the-art performance, but their performance is limited by the amount of quality labeled data they need for training. Scarcity of labeled data and non-transferablity of models, due to cross-domain discrepancy makes it a bigger challenge for remote sensing imagery analysis. In this work, we approach this problem through adversarial learning, driven by entropy and boundary of region-of-interest for unsupervised domain adaptation. This concept helps with better boundary prediction and encourages target domain entropy maps (probability/uncertainty maps) to be similar to source domains. In particular, we showed that deriving informative entropy through the adversarial learning is essential to enable the adaptation. We used a large scale cross country building extraction dataset to validate the framework. The experimental results show the usefulness of considering boundary and entropy driven adversarial learning for adaptation.

Co-registrating is a common pre-processing step for existing change detection algorithms, but registering bi-temporal images is nontrivial. The use of image patch as input for deep learning techniques provides a natural avenue to apply them in the OBIA framework, and have shown successful performance in the object-based land cover mapping and change detection applications. Even though attempts of applying deep learning techniques for change detection applications have been made with varying success, its application under OBIA framework for change detection have not been conducted and its tolerance for misregistration among temporal images are neither known. This study performed change detection under OBIA framework using deep learning techniques for the first time, and evaluated its performance regarding their tolerance of image misregistration on training and testing dataset. Our results demonstrate the proposed change detection scheme is surprisingly robust to image misregistration on the testing dataset, while classifiers trained with the training dataset containing image misregistration errors suffer from slight decrease of overall accuracy.

Most current state-of-the-art methods for semantic segmentation on remote sensing imagery require large labeled data, which is scarcely available. Due to the distribution shifting phenomenon inherent in remote sensing imagery, the reuse of pre-trained models on new areas of interest rarely yield satisfactory results. In this paper, we approach this problem from an adversarial learning perspective toward unsupervised domain adaptation. The core concept is to infuse fully convolutional neural networks and adversarial networks for semantic segmentation assuming the structures in the scene and objects of interest are similar in two set of images. Models are trained on a source dataset where ground truth is available and adapted to new target dataset iteratively via a adversarial loss on unlabeled samples. We use two real large scale datasets to validate the framework: 1) cross city road extraction and 2) cross country building extraction. The preliminary results show the usefulness of considering adversarial learning for indirect re-use of the pre-trained models. Experimental validation suggests significant benefits over models without adaptation.

Summary This paper presents a scalable object detection workflow for detecting objects, such as settlements, from remotely sensed (RS) imagery. We have successfully deployed this workflow on Titan supercomputer and utilized it for the task of mapping human settlement at a country scale. The performance of various stages in the workflow was analyzed before making it operational. The workflow implemented various strategies to address issues such as suboptimal resource utilization and long‐tail effects due to unbalanced image workload, data loss due to runtime failures, and maximum wall‐time constraints imposed by Titan's job scheduling policy. A mean shift clustering–based static load balancing strategy was implemented, which partitions the image load such that each partition contained similar‐sized images. Furthermore, a checkpoint‐restart strategy was added in the workflow as a fault‐tolerance mechanism to prevent the data losses due to unforeseen runtime failures. The performance of the above‐mentioned strategies was observed in various scenarios, such as node failure, exceeding wall time, and successful completion. Using this workflow, we have processed an RS data set that has a spatial resolution of 0.31 m and is comprised of 685 675 km2 of area of the Republic of Zambia in under six hours using 5426 nodes of the Titan supercomputer.

We test this premise and explore representation spaces from a single deep convolutional network and their visualization to argue for a novel unified feature extraction framework. The objective is to utilize and re-purpose trained feature extractors without the need for network retraining on three remote sensing tasks i.e. superpixel mapping, pixel-level segmentation and semantic based image visualization. By leveraging the same convolutional feature extractors and viewing them as visual information extractors that encode different settlement representation spaces, we demonstrate a preliminary inductive transfer learning potential on multiscale experiments that incorporate edge-level details up to semantic-level information.

Very large overhead imagery associated with ground truth maps has the potential to generate billions of training image patches for machine learning algorithms. However, random sampling selection criteria often leads to redundant and noisy-image patches for model training. With minimal research efforts behind this challenge, the current status spells missed opportunities to develop supervised learning algorithms that generalize over wide geographical scenes. In addition, much of the computational cycles for large scale machine learning are poorly spent crunching through noisy and redundant image patches. We demonstrate a potential framework to address these challenges specifically, while evaluating a human settlement detection task. A novel binary search tree sampling scheme is fused with a kernel based hashing procedure that maps image patches into hash-buckets using binary codes generated from image content. The framework exploits inherent redundancy within billions of image patches to promote mostly high variance preserving samples for accelerating algorithmic training and increasing model generalization.

Establishing up-to-date nationwide building maps is essential to understand urban dynamics, such as estimating population and urban planning and many other applications. However, an efficient and effective solution is yet to be developed. In this paper, for the first time we evaluate three state-of-the-art CNNs for detecting buildings across entire United States using aerial images. The three CNN architectures, fully convolutional neural network, conditional random field as recurrent neural network, and SegNet, support semantic pixel-wise labeling and focus on capturing textural information at multi-scale. We use 1-meter resolution NAIP images as the test data set, and compare the detection results across the three methods. In addition, we propose to combine signed distance function labels with SegNet, which is the preferred CNN architecture identified by our extensive evaluations. The results are further improved in terms of precision, recall rate and the number of building detected. On average, model inference on test images is less than one minute for an area of size ∼ 56 km2. With these promising results and the time required to process images, the framework offers great potential toward country scale building mapping with remote sensing imagery.

Up-to-date maps of installed solar photovoltaic panels are a critical input for policy and financial assessment of solar distributed generation. However, such maps for large areas are not available. With high coverage and low cost, aerial images enable large-scale mapping, but it is highly difficult to automatically identify solar panels from images, which are small objects with varying appearances dispersed in complex scenes. We introduce a new approach based on deep convolutional networks, which effectively learns to delineate solar panels in aerial scenes. The approach is applied to mapping solar panels in imagery covering 200 square kilometers in two cities, using only 12 square kilometers of training data that are manually labeled. Results are generated efficiently with an accuracy comparable to manual mapping, demonstrating the effectiveness and scalability of our approach.

The classification of hyperspectral and multimodal remote sensing data is affected by two key problems: the high dimensionality of the input data and the limited number of the labeled samples. In this letter, a multimetric learning approach that combines feature extraction and active learning (AL) is introduced to deal with these two issues simultaneously. In particular, distinct metrics are assigned to different types of features and then learned jointly. In this way, multiple features are projected into a common feature space, in which AL is then performed in conjunction with k- nearest neighbor classification to enrich the set of labeled samples. Experiments on two sets of remote sensing data illustrate the effectiveness of the proposed framework in terms of both classification accuracy and computational requirements.

Classification of remotely sensed hyperspectral images via supervised approaches is typically affected by high dimensionality of the spectral data and a limited number of labeled samples. Dimensionality reduction via feature extraction and active learning (AL) are two approaches that researchers have investigated independently to deal with these two problems. In this paper, we propose a new method in which the feature extraction and AL steps are combined into a unique framework. The idea is to learn and update a reduced feature space in a supervised way at each iteration of the AL process, thus taking advantage of the increasing labeled information provided by the user. In particular, the computation of the reduced feature space is based on the large-margin nearest neighbor (LMNN) metric learning principle. This strategy is applied in conjunction with k-nearest neighbor ( k-NN) classification, for which a new sample selection strategy is proposed. The methodology is validated experimentally on four benchmark hyperspectral data sets. Good improvements in terms of classification accuracy and computational time are achieved with respect to the state-of-the-art strategies that do not combine feature extraction and AL.

Multitemporal hyperspectral images provide valuable information for a wide range of applications related to supervised classification, including long-term environmental monitoring and land cover change detection. However, the required ground reference data are time-consuming and expensive to acquire, motivating researchers to investigate options for reusing limited training data for classification of other temporal images. Current studies that address high dimensionality and nonstationarity inherent in temporal hyperspectral data for classification are limited for the case where significant spectral drift exists between images. In this paper, we adapt and extend two manifold alignment (MA) methods for classification of multitemporal hyperspectral images in a common manifold space, assuming that the local geometries of two temporal spectral images are similar. The first method exploits a locally based manifold configuration of a source image (considered to be the “prior” manifold), and the second approach links local manifolds of two images using bridging pairs. In addition to exploiting manifolds estimated with spectral information for MA, we also demonstrate how spatial information can be incorporated into the MA methods. When evaluated using three Hyperion data sets, the proposed methods outperform four baseline approaches and two state-of-the-art domain adaptation methods. The advantages of the proposed MA methods are more evident when significant spectral drift exists between two temporal images. In addition to the promising classification results, the proposed methods establish a domain adaptation framework for analysis of temporal hyperspectral data based on data geometry.

Adapting a pre-trained classifier with labeled samples from an image for classification of another temporally related image is a common multitemporal image classification strategy. However, the adaptation is not effective when the spectral drift exhibited in temporal data is significant. Instead of iteratively redefining classifier parameters, we exploit similar data geometries of temporal data and project temporal data into a joint manifold space where similar samples are clustered. The proposed classification framework is based on aligning global temporal data manifolds. In addition to global structures, we also consider the local scale by incorporating local point relations into the alignment process. In experiments with challenging temporal hyperspectral data, the proposed framework provides favorable classification results, compared to the baseline.

Exploiting disparate features from potentially different data sources with multiple-kernel based machine learning is a promising approach for analyzing geo-spatial data. A mixture-of-kernel approach can facilitate construction of a more effective training data pool with Active Learning (AL). In addition, this could alleviate the computational burden in AL implementations. Kernel based learning requires hyperparameter tuning for model selection. Further, an optimal function is required to integrate different features or data sources appropriately in the kernel induced space. Both kernel parameters and kernel combination functions may need to be tuned at each AL learning step, which is potentially very time-consuming. In this paper, a novel multiple kernel active learning algorithm is proposed that promises enhanced classification, improved AL performance, and a mechanism for automatic selection of kernel weights in the mixture-of-kernels. We demonstrate the usefulness of the proposed framework with results for both feature fusion and sensor fusion tasks.

Active learning, which has a strong impact on processing data prior to the classification phase, is an active research area within the machine learning community, and is now being extended for remote sensing applications. To be effective, classification must rely on the most informative pixels, while the training set should be as compact as possible. Active learning heuristics provide capability to select unlabeled data that are the “most informative” and to obtain the respective labels, contributing to both goals. Characteristics of remotely sensed image data provide both challenges and opportunities to exploit the potential advantages of active learning. We present an overview of active learning methods, then review the latest techniques proposed to cope with the problem of interactive sampling of training pixels for classification of remotely sensed data with support vector machines (SVMs). We discuss remote sensing specific approaches dealing with multisource and spatially and time-varying data, and provide examples for high-dimensional hyperspectral imagery.

Similarity measures for classification of hyperspectral data in the manifold space are typically based on spectral characteristics. However, samples that are not spectrally separable may cause incorrectly connected graphs and result in noninformative data manifolds. Spatial relationships inherent in remote sensing images can be beneficial for constructing connectivity graphs. A spectral-spatial proximity graph utilizing both spectral characteristics and spatial homogeneity is proposed for robust manifold learning. With the proposed spectral-spatial graph, we are able to extract essential features and preserve important knowledge in a lower dimensional manifold space, where classification tasks can be performed effectively. Two hyperspectral data sets were used to validate the proposed approach. Classification results obtained by the nearest neighbor classifier demonstrate the usefulness of exploiting spectral similarity and spatial proximity for the manifold-based classification.

While spectral and temporal advantages of multitemporal hyperspectral images provide opportunities for advancing classification of time varying phenomena, significant challenges are associated with high dimensionality and nonstationary signatures. While manifold learning retains critical geometry and develops a low dimension space where class clusters are recovered, spectral changes in temporal imagery impact the fidelity of the geometric representation of class dependent data. In this paper, we investigate a manifold alignment framework that exploits prior information while exploring similar local structures. The aim is to make use of common underlying geometries of two multitemporal images and embed the resemblances in a joint data manifold for classification tasks. Promising results support the advantages of the proposed manifold alignment approach.

The wavelet-based feature extraction algorithms have been developed to explore the useful information for the hyperspectral image classification. On the other hand, the idea of using artificial neural network (ANNs) has also proved useful for hyperspectral image classification. To combine the advantages of ANNs with wavelet-based feature extraction methods, the wavelet network (WN) has been proposed for data identification and classification. The value of wavelet networks lies in their capabilities of extracting essential features in time-frequency plane. Both the position and the dilation of the wavelets are optimized besides the weights of the network during the training phase. In this paper, the basic concept of wavelet-based feature extraction is firstly described. Then the theory of wavelet networks is introduced for the hyperspectral image classification. Finally an AVIRIS image was used to test the feasibility and performance of classification using the wavelet networks. The experiment results showed that the wavelet networks exactly an effective tool for classification of hyperspectral images, and have better classification results than the traditional feed-forward multilayer neural networks.