Clouds pose the most significant hindrance to satellite imagery analysis. A crucial step in pre-processing satellite imagery involves masking these clouds, which ensures reliable downstream geospatial analysis. However, this masking also reduces the effectively analyzed region. To address this, recent studies have explored imputing missing values under clouds to restore a complete image. Unfortunately, the inaccuracy of the off-the-shelf masks (e.g., QA band from Landsat 8) adversely affects downstream geospatial analysis and machine learning tasks. Accurate imputation necessitates precise detection of clouds, including thin clouds and shadows. Of recent cloud segmentation models segmenting a subset of the cloud phenomena (i.e., thick clouds, thin clouds, and cloud-shadows), their accuracy, though promising, remains insufficient; even fewer models segment all three phenomena with adequate performance. Foundational models show promise for image segmentation tasks, but studies with them in cloud segmentation are sparse. In this paper, we present a thorough experimental analysis, studying different properties of foundational model and their effectiveness on this comprehensive task at different levels of transfer learning. This analysis will explore foundational model architecture, pretraining dataset, and scheme. The main result we found is the transformer architecture demonstrates superior performance on granular details on boundaries and small-sized segments, compared to convolutional or hybrid architectures.

Recent advancements in remote sensing technology have led to vast collections of big image data, ranging from high-resolution (sub-meter) to medium-resolution (10 to 30 meters). While medium-resolution images are freely available, high-resolution imagery is costly. Several applications, such as object recognition and cloud imputation, necessitate super-resolving medium-resolution images to align with high-resolution data. Deep learning has significantly improved single-image super-resolution (SISR) accuracy. However, super-resolving at scales exceeding 5x (10 meters to 2 or 1 meter) remains challenging. Unlike natural photographs, remote sensing images require recovering intricate details for 25 pixels at a super-resolution of 2 meters from an aggregated single pixel at a medium resolution of 10 meters. This demands the ability to discern subtle geographic features, such as textures, variations in vegetation and land use changes, and small man-made structures. Existing SR techniques, primarily designed and tested on natural images, focus on establishing hidden pixel relationships by learning contextual constraints from high-resolution data and employing innovative loss functions during high-resolution image reconstruction through residual or similarity transformations. However, their effectiveness with geospatial data requires careful evaluation. To evaluate the effectiveness of various super-resolution (SR) techniques, this paper conducts a comparative study using a diverse dataset of over 16,530 high-resolution satellite images from Planet Explorer’s remote sensing database, sourced from Sentinel-2 (10-meter) and RapidEye (3-meter) satellites. The performance of each method is assessed and compared using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) metrics. We hope the insights from this comparative study will help guide the selection of suitable SR methods by highlighting their advantages and disadvantages.

Multi-sensor spatiotemporal satellite images have become crucial for monitoring the geophysical characteristics of the Earth’s environment. However, clouds often obstruct the view from the optical sensors mounted on satellites and therefore degrade the quality of spectral, spatial, and temporal information. Though cloud imputation with the rise of deep learning research has provided novel ways to reconstruct the cloud-contaminated regions, many learning-based methods still lack the capability of harmonizing the differences between similar spectral bands across multiple sensors. To cope with the inter-sensor inconsistency of overlapping bands in different optical sensors, we propose a novel harmonization-guided residual network to impute the areas under clouds. We present a knowledge-guided harmonization model that maps the reflectance response from one satellite collection to another based on the spectral distribution of the cloud-free pixels. The harmonized cloud-free image is subsequently exploited in the intermediate layers as an additional input, paired with a custom loss function that considers image reconstruction quality and inter-sensor consistency jointly during training. To demonstrate the performance of our model, we conducted extensive experiments on a multi-sensor remote sensing imagery benchmark dataset consisting of widely used Landsat-8 and Sentinel-2 images. Compared to the state-of-the-art methods, results show at least a 22.35% improvement in MSE.

Huge amounts of remote sensing data collected from hundreds of operational satellites in conjunction with on-demand UAV based imaging products are offering unprecedented capabilities towards monitoring dynamic earth resources. However, searching for the right combination of imagery products that satisfy an application requirement is a daunting task. Earlier efforts at streamlining remote sensing data discovery include NASA’s Earth Observing System (EOS) Data and Information System (EOSDIS), USGS Global Visualization Viewer (GloVis), and several other research systems like Minnesota MapServer. These systems were built on top of metadata harvesting, indexing, keyword searching modules which were not scalable and interoperable. To address these challenges, recently the SpatioTemporal Asset Catalog (STAC) specification was developed to provide a common language to describe a range of geospatial information, so that data products can be more easily indexed and discovered. In this paper we present an highly scalable STAC API based system with spatiotemporal indexing support. Experimental evaluation shows that our spatiotemporal indexing based queries are 1000x faster than standard STAC API server.

The Food and Agriculture Organization (FAO) of the United Nations predicts that in order to meet the needs of the expected 3 billion population growth by 2050, food production has to increase by 60%. Therefore, monitoring and mapping crops accurately is essential for estimating food production during each crop growing season across the globe. Traditionally, multispectral remote sensing imagery has been widely used for mapping crops worldwide. However, single date imagery does not capture temporal characteristics (phenology) of growing crops, leading to imprecise crop maps and food estimates. On the other hand, purely temporal classification approaches also produce inaccurate crop maps as they do not account for spatial autocorrelations. In this paper, we present a multimodal deep learning solution that jointly exploits spatial-spectral and phenological properties to identify major crop types. Using a two stream architecture, spatial characteristics are captured via a spatial stream consisting of very high resolution images (single date, 1m, 3-spectral bands, USDA NAIP) with a CNN and the phenological characteristics via a temporal stream images (biweekly, 250m, MODIS NDVI) with an LSTM. Experimental results show that the proposed multimodal solution reduces prediction error by 60%.

Geosimulations using computer simulation models provideGI scientists an effective way to study complex geographic phenomena and predict future outcomes. Typically, geosimulations are developed to execute in an HPC environment with parallel and distributed execution capabilities. However, traditional HPC environments limit these simulations to a static runtime environment, where resources for execution must be decided before execution. Traditional simulation approaches such as a data parallel approach assigns fixed computing resources on every unit of data (e.g., a tile or a county). However, in many practical situations, a user may want to assign additional computing resources to speedup or perform more computation in a specific region. For example, in an urban growth model (UGM) simulation, to explore the outcomes of changes due to urban policy in a tile or a group of tiles at a given time-step, an urban geographer may want to assign more computing resources to those group of tiles to quickly determine impacts of policy on urbanization. In the absence of a dynamic resource allocation mechanism, the utility of a geosimulation to explore what-if scenarios on-the-fly is limited to pre-allocated computing resources. Thus, to effectively leverage existing resources, we first design a co-scheduling approach for geosimulations in a resource constrained HPC environment. We then present a second design for a geosimulation which allows dynamic provisioning of resources in an HPC environment based on run-time users' demands. Finally, to demonstrate the utility of the two approaches we modify the FUTURES geosimulation to support computationally expensive high-resolution simulation in regions of interest (ROIs) as specified by a user using the FUTURES-DPE framework.

Geographic modeling using geosimulations is a popular approach to explore outcomes from interacting geographic processes in a region. Geosimulations account for space, time, and complex spatial and spatiotemporal relationships to explore "what-if" scenarios and their potential impact in a region. However, current approaches to geosimulation limit manipulating model input and exploring alternative scenarios by controlling the simulation at runtime. Furthermore, lack of runtime support for visualization hinders the ability to view the current state of a simulation to provide meaningful steering input. In this paper, we propose a computational steering system for geosimulations, called tFUTURES, that allows users to specify steering input and execute steering actions at runtime. The core of the proposed system includes: (i) Visualization Service that provides a minimal web-based user interface; (ii) Monitoring Server that receives and handles user-initiated steering actions; and (iii) Steering Client that executes the steering actions by altering the control flow of the geosimulation at runtime. Further, we develop versioning and checkpointing features for the system to support: (i) concurrent execution paths of a geosimulation based on varying inputs in a time-step; and (ii) controlled execution with the ability to pause, advance or rollback a geosimulation. To evaluate our computational steering system, we modify the FUTURES Urban Growth Model (UGM) geosimulation to support user-initiated steering input and steering actions from a web browser. Experimental results demonstrate minimal system overhead with observed end-to-end steering latency of about 5 and 11 seconds for a single time-step of the simulation when measured in a local and distributed computing environment, respectively.

Monitoring biomass over large geographic regions for changes in vegetation and cropping patterns is important for many applications. Changes in vegetation happen due to reasons ranging from climate change and damages to new government policies and regulations. Remote sensing imagery (multi-spectral and multi-temporal) is widely used in change pattern mapping studies. Existing bi-temporal change detection techniques are better suited for multi-spectral images and time series based techniques are more suited for analyzing multi-temporal images. A key contribution of this work is to define change as hierarchical rather than boolean. Based on this definition of change pattern, we developed a novel time series similarity based change detection framework for identifying inter-annual changes by exploiting phenological properties of growing crops from satellite time series imagery. The proposed framework consists of three components: hierarchical clustering tree construction, nearest neighbor based classification, and change detection using similarity hierarchy. Though the proposed approach is unsupervised, we present evaluation using manually induced change regions embedded in the real dataset. We compare our method with the widely used K-Means clustering and evaluation shows that K-Means over-detects changes in comparison to our proposed method.

Urban neighborhood classification using very high resolution (VHR) remote sensing imagery is a challenging and {\em emerging} application. A semi-supervised learning approach for identifying neighborhoods is presented which employs superpixel tessellation representations of VHR imagery. The image representation utilizes homogeneous and irregularly shaped regions termed superpixels and derives novel features based on intensity histograms, geometry, corner and superpixel density and scale of tessellation. The semi-supervised learning approach uses a support vector machine (SVM) to obtain a preliminary classification which is then subsequently refined using graph Laplacian propagation. Several intermediate stages in the pipeline are presented to showcase the important features of this approach. We evaluated this approach on four different geographic settings with varying neighborhood types and compared it with the recent Gaussian Multiple Learning algorithm. This evaluation shows several advantages, including model building, accuracy, and efficiency which makes it a great choice for deployment in large scale applications like global human settlement mapping and population distribution (e.g., LandScan), and change detection.

In this paper, we present a computationally efficient algorithm based on multiple instance learning for mapping informal settlements (slums) using very high-resolution remote sensing imagery. From remote sensing perspective, informal settlements share unique spatial characteristics that distinguish them from other urban structures like industrial, commercial, and formal residential settlements. However, regular pattern recognition and machine learning methods, which are predominantly single-instance or per-pixel classifiers, often fail to accurately map the informal settlements as they do not capture the complex spatial patterns. To overcome these limitations we employed a multiple instance based machine learning approach, where groups of contiguous pixels (image patches) are modeled as generated by a Gaussian distribution. We have conducted several experiments on very high-resolution satellite imagery, representing four unique geographic regions across the world. Our method showed consistent improvement in accurately identifying informal settlements.

Online time series change detection is a critical component of many monitoring systems, such as space and air-borne remote sensing instruments, cardiac monitors, and network traffic profilers, which continuously analyze observations recorded by sensors. Data collected by such sensors typically has a periodic component. Most existing time series change detection methods are not directly applicable to handle such data, either because they are not designed to handle periodic time series or because they cannot operate in an online mode. We propose an online change detection algorithm which can handle periodic time series. The algorithm uses a Gaussian process based non-parametric time series prediction model and monitors the difference between the predictions and actual observations within a statistical control chart framework to identify changes. A key challenge in using Gaussian process in an online mode is the need to solve a large system of equations involving the associated covariance matrix which grows with every time step. The proposed algorithm exploits the special structure of the covariance matrix and can analyze a time series of length T in O(T2) time while maintaining a O(T) memory footprint, compared to O(T4) time and O(T2) memory requirement of standard matrix manipulation methods. We experimentally demonstrate the superiority of the proposed algorithm over several existing time series change detection algorithms on a set of synthetic and real time series. Finally, we illustrate the effectiveness of the proposed algorithm for identifying land use land cover changes using Normalized Difference Vegetation Index (NDVI) data collected for an agricultural region in Iowa state, USA. Our algorithm is able to detect different types of changes in a NDVI validation data set (with ≈ 80% accuracy) which occur due to crop type changes as well as natural disasters.

We demonstrate a Google Earth based novel visualization system for continuous monitoring of biomass at regional and global scales. This system is integrated with a back-end spatiotemporal data mining system that continuously detects changes using high temporal resolution MODIS images. In addition to the visualization, we demonstrate novel query features of the system that provides insights into the current conditions of the landscape.

Intelligent image information mining for thematic pattern extraction is a complex task. Ever increasing spatial, spectral, and temporal resolution poses several challenges to the geographic knowledge discovery community. Although the improvements in sensor technology and data collection methods may lead to improved geoinformation generation, it also places several constraints on data mining techniques. Moreover thematic classes are spectrally overlapping, that is, many thematic classes can not be separated by spectral features alone. In recent years we have developed several innovative machine learning approaches to address these problems. The resulting software system, called *Miner, was tested on several real world multisource spatiotemporal datasets. Experimental evaluation showed improved accuracy over conventional data mining approaches. In addition, we integrated *Miner with another popular open source machine learning system called Weka. In this demo we show the utility of *Miner for thematic information extraction from multisource spatiotemporal data (remote sensing images and ancillary geospatial databases).

The integration of disparate heterogeneous spatial databases for extending queries is a challenging task. The authors present a novel framework, based on a k-nearest neighbor (kNN) algorithm, for integrating remote sensing imagery with Forest Inventory Analysis (FIA) sample point/plot data managed in a relational database system. We then demonstrate how queries to this system may be extended over any arbitrary region of interest in a Web based geographical information system. To build the integrated database, spectral signatures are collected at FIA plot locations from the Landsat TM image. A plot-id image is produced by assigning each pixel to the closest FIA plot in multi-dimensional spectral space. The resulting image provides an interface to the Forest Inventory Analysis Data-Base (FIADB) and allows generalizations of the estimates for any user defined query window or region of interest (ROI). This methodology, along with geostatistical analysis, is integrated into a client/server Web based geographical information system, which provides Internet users with an easy to use query interface for the FIADB and spatial databases.