Clouds pose the most significant hindrance to satellite imagery analysis. A crucial step in pre-processing satellite imagery involves masking these clouds, which ensures reliable downstream geospatial analysis. However, this masking also reduces the effectively analyzed region. To address this, recent studies have explored imputing missing values under clouds to restore a complete image. Unfortunately, the inaccuracy of the off-the-shelf masks (e.g., QA band from Landsat 8) adversely affects downstream geospatial analysis and machine learning tasks. Accurate imputation necessitates precise detection of clouds, including thin clouds and shadows. Of recent cloud segmentation models segmenting a subset of the cloud phenomena (i.e., thick clouds, thin clouds, and cloud-shadows), their accuracy, though promising, remains insufficient; even fewer models segment all three phenomena with adequate performance. Foundational models show promise for image segmentation tasks, but studies with them in cloud segmentation are sparse. In this paper, we present a thorough experimental analysis, studying different properties of foundational model and their effectiveness on this comprehensive task at different levels of transfer learning. This analysis will explore foundational model architecture, pretraining dataset, and scheme. The main result we found is the transformer architecture demonstrates superior performance on granular details on boundaries and small-sized segments, compared to convolutional or hybrid architectures.

Recent advancements in remote sensing technology have led to vast collections of big image data, ranging from high-resolution (sub-meter) to medium-resolution (10 to 30 meters). While medium-resolution images are freely available, high-resolution imagery is costly. Several applications, such as object recognition and cloud imputation, necessitate super-resolving medium-resolution images to align with high-resolution data. Deep learning has significantly improved single-image super-resolution (SISR) accuracy. However, super-resolving at scales exceeding 5x (10 meters to 2 or 1 meter) remains challenging. Unlike natural photographs, remote sensing images require recovering intricate details for 25 pixels at a super-resolution of 2 meters from an aggregated single pixel at a medium resolution of 10 meters. This demands the ability to discern subtle geographic features, such as textures, variations in vegetation and land use changes, and small man-made structures. Existing SR techniques, primarily designed and tested on natural images, focus on establishing hidden pixel relationships by learning contextual constraints from high-resolution data and employing innovative loss functions during high-resolution image reconstruction through residual or similarity transformations. However, their effectiveness with geospatial data requires careful evaluation. To evaluate the effectiveness of various super-resolution (SR) techniques, this paper conducts a comparative study using a diverse dataset of over 16,530 high-resolution satellite images from Planet Explorer’s remote sensing database, sourced from Sentinel-2 (10-meter) and RapidEye (3-meter) satellites. The performance of each method is assessed and compared using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) metrics. We hope the insights from this comparative study will help guide the selection of suitable SR methods by highlighting their advantages and disadvantages.

Recent advances in machine learning and deep learning have led to the widespread use of Conversational AI in many practical applications. However, it is still very challenging to leverage auxiliary information that can provide conversational context or personalized tuning to improve the quality of conversations. For example, there has only been limited research on using an individual’s persona information to improve conversation quality, and even state-of-the-art conversational AI techniques are unable to effectively leverage signals from heterogeneous sources of auxiliary data, such as multi-modal interaction data, demographics, SDOH data, etc. In this paper, we present a novel Persona-Coded Poly-Encoder method that leverages persona information in a multi-stream encoding scheme to improve the quality of response generation for conversations. To show the efficacy of the proposed method, we evaluate our method on two different persona-based conversational datasets, and compared against two state-of-the-art methods. Our experimental results and analysis demonstrate that our method can improve conversation quality over the baseline method Poly-Encoder by $ 3.32\%$ and $ 2.94\%$ in terms of BLEU score and HR@1, respectively. More significantly, our method offers a path to better utilization of multi-modal data in conversational tasks. Lastly, our study outlines several challenges and future research directions for advancing personalized conversational AI technology.

Many downstream applications, such as agricultural monitoring, damage assessments, and urban planning, have benefited from remote sensing imagery. Ortho image generation from very high-resolution images has unpleasant side effects in places of building occlusions leaving holes in the orthorectified images. As a result, blank pixels caused by orthorectification must be filled in prior to downstream tasks such as machine learning. In this paper, we present a new deep learning architecture based on Wide Activation Super Resolution (WDSR) network combined with an Adaptive Instance Normalization (AdaIN) based style transfer for imputing holes (missing pixels) in orthorectified images. To test and validate the performance of imputation algorithms, we developed a new multi-resolution benchmark dataset consisting of WorldView-3 and Sentinel-2 images. Our experiments show that the WDSR framework outperforms other state-of-the-art (SOTA) deep learning methods and the Ordinary Kriging method. Our proposed method has improved the mean squared error (MSE) by at least 12.54% in comparison to existing SOTA methods.

Multi-sensor spatiotemporal satellite images have become crucial for monitoring the geophysical characteristics of the Earth’s environment. However, clouds often obstruct the view from the optical sensors mounted on satellites and therefore degrade the quality of spectral, spatial, and temporal information. Though cloud imputation with the rise of deep learning research has provided novel ways to reconstruct the cloud-contaminated regions, many learning-based methods still lack the capability of harmonizing the differences between similar spectral bands across multiple sensors. To cope with the inter-sensor inconsistency of overlapping bands in different optical sensors, we propose a novel harmonization-guided residual network to impute the areas under clouds. We present a knowledge-guided harmonization model that maps the reflectance response from one satellite collection to another based on the spectral distribution of the cloud-free pixels. The harmonized cloud-free image is subsequently exploited in the intermediate layers as an additional input, paired with a custom loss function that considers image reconstruction quality and inter-sensor consistency jointly during training. To demonstrate the performance of our model, we conducted extensive experiments on a multi-sensor remote sensing imagery benchmark dataset consisting of widely used Landsat-8 and Sentinel-2 images. Compared to the state-of-the-art methods, results show at least a 22.35% improvement in MSE.

Detecting changes in real-time using remote sensing data is of paramount importance in areas such as crop health monitoring, weed detection, and disaster management. However, real-time change detection using remote sensing imagery faces several challenges: a) it requires real-time data extraction which is a challenge for traditional satellite imagery sources such as MODIS and LANDSAT due to the latency associated with collecting and processing the data. Due to the advances made in the past decade in drone technology, Unmanned Aerial Vehicles (UAVs) can be used for real-time data collection. However, a large percentage of this data will be unlabeled which limits the use of well-known supervised machine learning methods; b) from an infrastructure perspective, the cloud-edge solution of processing the data collected from UAVs (edge) only on the cloud is also constrained by latency and bandwidth-related issues. Due to these limitations, transferring large amounts of data between cloud and edge, or storing large amounts of information regarding past time periods on an edge device is infeasible. We can limit the amount of data transferred between the cloud and edge by performing analyses on-the-fly at the edge using low-power devices (edge devices) that can be connected to UAVs. However, edge devices have computational and memory bottlenecks, which would limit the usage of complex machine learning algorithms. In this paper, we demonstrate how an unsupervised GMM-based real-time change detection method at the edge can be used to identify weeds in real-time. We evaluate the scalability of our method on edge computing and traditional devices such as NVIDIA Jetson TX2, RTX 2080, and traditional Intel CPUs. We perform a case study for weed detection on images collected from UAVs. Our results demonstrate both the efficacy and computational efficiency of our method.

For more than five decades, remote sensing imagery has been providing critical information for many applications such as crop monitoring, disaster assessment, and urban planning. Unfortunately, more than 50% of optical remote sensing images are contaminated by clouds severely affecting the object identification. However, thanks to recent advances in remote sensing instruments and increase in number of operational satellites, we now have petabytes of multi-sensor observations covering the globe. Historically cloud imputation techniques were designed for single sensor images, thus existing benchmarks were mostly limited to single sensor images, which precludes design and validation of cloud imputation techniques on multi-sensor data. In this paper, we introduce a new benchmark data set consisting of images from two widely used and publicly available satellite images, Landsat-8 and Sentinel-2, and a new multi-stream deep residual network (MDRN). This newly introduced benchmark dataset fills an important gap in the existing benchmark datasets, which allows exploitation of multi-resolution spectral information from the cloud-free regions of temporally nearby images, and the MDRN algorithm addresses imputation using the multi-resolution data. Both quantitative and qualitative experiments show that the utility of our benchmark dataset and as well as efficacy of our MDRN architecture in cloud imputation. The MDRN outperforms the closest competing method by 14.1%.

Earth observations from remote sensing imagery play an important role in many environmental applications ranging from natural resource (e.g., crops, forests) monitoring to man-made object (e.g., builds, factories) recognition. Most widely used optical remote sensing data however is often contaminated by clouds making it hard to identify the objects underneath. Fortunately, with the recent advances and increased operational satellites, the spatial and temporal density of image collections have significantly increased. In this paper, we present a novel deep learning-based imputation technique for inferring spectral values under the clouds using nearby cloud-free satellite image observations. The proposed deep learning architecture, extended contextual attention (ECA), exploits similar properties from the cloud-free areas to tackle clouds of different sizes occurring at arbitrary locations in the image. A contextual attention mechanism is incorporated to utilize the useful cloud-free information from multiple images. To maximize the imputation performance of the model on the cloudy patches instead of the entire image, a two-phase custom loss function is deployed to guide the model. To study the performance of our model, we trained our model on a benchmark Sentinel-2 dataset by superimposing real-world cloud patterns. Extensive experiments and comparisons against the state-of-the-art methods using pixel-wise and structural metrics show the improved performance of our model. Our experiments demonstrated that the ECA method is consistently better than all other methods, it is 28.4% better on MSE and 31.7% better on cloudy MSE as compared to the state-of-the-art EDSR network.

This article summarizes research trends on the topic of anomaly detection in video feeds of a single scene. We discuss the various problem formulations, publicly available datasets and evaluation criteria. We categorize and situate past research into an intuitive taxonomy and provide a comprehensive comparison of the accuracy of many algorithms on standard test sets. Finally, we also provide best practices and suggest some possible directions for future research.

Huge amounts of remote sensing data collected from hundreds of operational satellites in conjunction with on-demand UAV based imaging products are offering unprecedented capabilities towards monitoring dynamic earth resources. However, searching for the right combination of imagery products that satisfy an application requirement is a daunting task. Earlier efforts at streamlining remote sensing data discovery include NASA’s Earth Observing System (EOS) Data and Information System (EOSDIS), USGS Global Visualization Viewer (GloVis), and several other research systems like Minnesota MapServer. These systems were built on top of metadata harvesting, indexing, keyword searching modules which were not scalable and interoperable. To address these challenges, recently the SpatioTemporal Asset Catalog (STAC) specification was developed to provide a common language to describe a range of geospatial information, so that data products can be more easily indexed and discovered. In this paper we present an highly scalable STAC API based system with spatiotemporal indexing support. Experimental evaluation shows that our spatiotemporal indexing based queries are 1000x faster than standard STAC API server.

The Mean Teacher (MT) model of Tarvainen and Valpola has shown good performance on several semi-supervised benchmark datasets. MT maintains a teacher model's weights as the exponential moving average of a student model's weights and minimizes the divergence between their probability predictions under diverse perturbations of the inputs. However, MT is known to suffer from confirmation bias, that is, reinforcing incorrect teacher model predictions. In this work, we propose a simple yet effective method called Local Clustering (LC) to mitigate the effect of confirmation bias. In MT, each data point is considered independent of other points during training; however, data points are likely to be close to each other in feature space if they share similar features. Motivated by this, we cluster data points locally by minimizing the pairwise distance between neighboring data points in feature space. Combined with a standard classification cross-entropy objective on labeled data points, the misclassified unlabeled data points are pulled towards high-density regions of their correct class with the help of their neighbors, thus improving model performance. We demonstrate on semi-supervised benchmark datasets SVHN and CIFAR-10 that adding our LC loss to MT yields significant improvements compared to MT and performance comparable to the state of the art in semi-supervised learning11The code is available at: https://github.com/jay1204/local_clustering_with_mt_for_ssl.

The Food and Agriculture Organization (FAO) of the United Nations predicts that in order to meet the needs of the expected 3 billion population growth by 2050, food production has to increase by 60%. Therefore, monitoring and mapping crops accurately is essential for estimating food production during each crop growing season across the globe. Traditionally, multispectral remote sensing imagery has been widely used for mapping crops worldwide. However, single date imagery does not capture temporal characteristics (phenology) of growing crops, leading to imprecise crop maps and food estimates. On the other hand, purely temporal classification approaches also produce inaccurate crop maps as they do not account for spatial autocorrelations. In this paper, we present a multimodal deep learning solution that jointly exploits spatial-spectral and phenological properties to identify major crop types. Using a two stream architecture, spatial characteristics are captured via a spatial stream consisting of very high resolution images (single date, 1m, 3-spectral bands, USDA NAIP) with a CNN and the phenological characteristics via a temporal stream images (biweekly, 250m, MODIS NDVI) with an LSTM. Experimental results show that the proposed multimodal solution reduces prediction error by 60%.

This work introduces a new approach to localize anomalies in surveillance video. The main novelty is the idea of using a Siamese convolutional neural network (CNN) to learn a distance function between a pair of video patches (spatio-temporal regions of video). The learned distance function, which is not specific to the target video, is used to measure the distance between each video patch in the testing video and the video patches found in normal training video. If a testing video patch is not similar to any normal video patch then it must be anomalous. We compare our approach to previously published algorithms using 4 evaluation measures and 3 challenging target benchmark datasets. Experiments show that our approach either surpasses or performs comparably to current state-of-the-art methods.

Anomaly detection on large scale spatio-temporal data such as climate data is a challenging task depending on the spatial and temporal resolution and autocorrelation of the data. When considering global gridded daily temperature data, the number of locations and the length of time period considered makes anomaly detection a big data problem. Gaussian Process (GP) Learning is a method that is well-suited to identify the complex spatial and temporal autocorrelation properties of spatio-temporal data. One of the primary challenges with using GP is the computational complexity associated with inverting a covariance matrix. This is further compounded when considering data on a national/global scale and performing anomaly detection using such methods often requires dedicated high performance computing platforms. In this paper, we describe a purely temporal scalable anomaly detection technique for gridded temperature data based on GP Learning that ignore the spatial autocorrelation between neighboring grids and perform anomaly detection on each of the grids in parallel, thereby reducing the execution time. We introduce three methods: a standalone data parallel approach using a single GPU, a distributed memory version on multi-node clusters using MPI, and a mixed parallel approach using multiple GPUs. In comparison to a sequential approach, they are 17.2x, 47.1x, and 88.9x faster, respectively.

Nuclear proliferation monitoring using remote sensing imagery is an essential area of research from a national security perspective. Studies have shown that traditional per-pixel and small patch based classification schemes prove to be insufficient in detecting complex facilities such as coal, gas and nuclear. In this paper, we evaluate the state of the art deep learning architectures for detecting the aforementioned complex facilities. We use the principles of transfer learning to fine-tune the state of the art deep learning architectures using models pre-trained on 1000 classes of Imagenet data. We evaluated the classification performance of these models on a benchmark imagery database consisting of two hundred complex facilities collected over the United States.

In this study, we explore deep learning based classification frameworks for state-wide crop classification using one meter resolution remote sensing imagery. We also studied the advantages of new Xeon processors (Xeon Gold 6134) in scaling deep learning models. Our study shows that deep learning models can indeed be scaled on CPUs, and can provide an easier solution over complex GPU implementations.

Geosimulations using computer simulation models provideGI scientists an effective way to study complex geographic phenomena and predict future outcomes. Typically, geosimulations are developed to execute in an HPC environment with parallel and distributed execution capabilities. However, traditional HPC environments limit these simulations to a static runtime environment, where resources for execution must be decided before execution. Traditional simulation approaches such as a data parallel approach assigns fixed computing resources on every unit of data (e.g., a tile or a county). However, in many practical situations, a user may want to assign additional computing resources to speedup or perform more computation in a specific region. For example, in an urban growth model (UGM) simulation, to explore the outcomes of changes due to urban policy in a tile or a group of tiles at a given time-step, an urban geographer may want to assign more computing resources to those group of tiles to quickly determine impacts of policy on urbanization. In the absence of a dynamic resource allocation mechanism, the utility of a geosimulation to explore what-if scenarios on-the-fly is limited to pre-allocated computing resources. Thus, to effectively leverage existing resources, we first design a co-scheduling approach for geosimulations in a resource constrained HPC environment. We then present a second design for a geosimulation which allows dynamic provisioning of resources in an HPC environment based on run-time users' demands. Finally, to demonstrate the utility of the two approaches we modify the FUTURES geosimulation to support computationally expensive high-resolution simulation in regions of interest (ROIs) as specified by a user using the FUTURES-DPE framework.

Geographic modeling using geosimulations is a popular approach to explore outcomes from interacting geographic processes in a region. Geosimulations account for space, time, and complex spatial and spatiotemporal relationships to explore "what-if" scenarios and their potential impact in a region. However, current approaches to geosimulation limit manipulating model input and exploring alternative scenarios by controlling the simulation at runtime. Furthermore, lack of runtime support for visualization hinders the ability to view the current state of a simulation to provide meaningful steering input. In this paper, we propose a computational steering system for geosimulations, called tFUTURES, that allows users to specify steering input and execute steering actions at runtime. The core of the proposed system includes: (i) Visualization Service that provides a minimal web-based user interface; (ii) Monitoring Server that receives and handles user-initiated steering actions; and (iii) Steering Client that executes the steering actions by altering the control flow of the geosimulation at runtime. Further, we develop versioning and checkpointing features for the system to support: (i) concurrent execution paths of a geosimulation based on varying inputs in a time-step; and (ii) controlled execution with the ability to pause, advance or rollback a geosimulation. To evaluate our computational steering system, we modify the FUTURES Urban Growth Model (UGM) geosimulation to support user-initiated steering input and steering actions from a web browser. Experimental results demonstrate minimal system overhead with observed end-to-end steering latency of about 5 and 11 seconds for a single time-step of the simulation when measured in a local and distributed computing environment, respectively.

In this paper, we present a nearest neighbor based hierarchical change detection methodology for analyzing multi-temporal remote sensing imagery. A key contribution of this work is to define change as hierarchical rather than boolean. Based on this definition of change pattern, we developed a novel time series similarity based change detection framework for identifying inter-annual changes by exploiting phenological properties of growing crops from satellite time series imagery. The proposed framework consists of four components: hierarchical clustering tree construction, nearest neighbor based classification, relaxation labeling, and change detection using similarity hierarchy. Though the proposed approach is unsupervised, we present evaluation using manually induced change regions embedded in the real dataset. We compare our method with the widely used K-Means clustering and evaluation shows that K-Means over-detects changes in comparison to our proposed method.

Increasing population, rapid urbanization, quest for biofuels, pollution, diseases, and adverse climate changes are some of the major drivers behind the changing surface of our planet. Timely monitoring and assessment of these changes, along with dissemination of accurate information, is important for policy makers, city planners, and humanitarian relief workers. Advances in remote sensing technologies have led to acquisition of very high resolution remote sensing imagery in the past decade. This data is highly useful for the aforementioned applications, and machine learning technology can be used to identify and quantify the changed regions. In this study we explore a semi-supervised deep generative model for change detection in very high resolution multispectral and bitemporal imagery. We constructed an auxiliary variational autoencoder that infers class labels without incurring high sample complexity costs. The resulting classifier was able to produce accurate predictions of real changes over images that appear significantly different due to environmental conditions (not real changes) while utilizing only a small set of labeled samples.

Monitoring biomass over large geographic regions for changes in vegetation and cropping patterns is important for many applications. Changes in vegetation happen due to reasons ranging from climate change and damages to new government policies and regulations. Remote sensing imagery (multi-spectral and multi-temporal) is widely used in change pattern mapping studies. Existing bi-temporal change detection techniques are better suited for multi-spectral images and time series based techniques are more suited for analyzing multi-temporal images. A key contribution of this work is to define change as hierarchical rather than boolean. Based on this definition of change pattern, we developed a novel time series similarity based change detection framework for identifying inter-annual changes by exploiting phenological properties of growing crops from satellite time series imagery. The proposed framework consists of three components: hierarchical clustering tree construction, nearest neighbor based classification, and change detection using similarity hierarchy. Though the proposed approach is unsupervised, we present evaluation using manually induced change regions embedded in the real dataset. We compare our method with the widely used K-Means clustering and evaluation shows that K-Means over-detects changes in comparison to our proposed method.

Graph data management and mining became a popular area of research, and led to the development of plethora of systems in recent years. Unfortunately, a number of emerging graph analysis systems assume different graph data models, and support different query interface and serialization formats. Such diversity, combined with a lack of comparisons, makes it complicated to understand the trade-offs between different systems and the graph operations for which they are designed. This study presents an evaluation of graph pattern matching capabilities of six graph analysis systems, by extending the Lehigh University Benchmark to investigate the degree of effectiveness to perform the same operation over the same graph in various graph analysis systems. Through the evaluation, this study reveals both quantitative and qualitative findings.

Urban neighborhood classification using very high resolution (VHR) remote sensing imagery is a challenging and {\em emerging} application. A semi-supervised learning approach for identifying neighborhoods is presented which employs superpixel tessellation representations of VHR imagery. The image representation utilizes homogeneous and irregularly shaped regions termed superpixels and derives novel features based on intensity histograms, geometry, corner and superpixel density and scale of tessellation. The semi-supervised learning approach uses a support vector machine (SVM) to obtain a preliminary classification which is then subsequently refined using graph Laplacian propagation. Several intermediate stages in the pipeline are presented to showcase the important features of this approach. We evaluated this approach on four different geographic settings with varying neighborhood types and compared it with the recent Gaussian Multiple Learning algorithm. This evaluation shows several advantages, including model building, accuracy, and efficiency which makes it a great choice for deployment in large scale applications like global human settlement mapping and population distribution (e.g., LandScan), and change detection.

In this paper, we present a computationally efficient algorithm based on multiple instance learning for mapping informal settlements (slums) using very high-resolution remote sensing imagery. From remote sensing perspective, informal settlements share unique spatial characteristics that distinguish them from other urban structures like industrial, commercial, and formal residential settlements. However, regular pattern recognition and machine learning methods, which are predominantly single-instance or per-pixel classifiers, often fail to accurately map the informal settlements as they do not capture the complex spatial patterns. To overcome these limitations we employed a multiple instance based machine learning approach, where groups of contiguous pixels (image patches) are modeled as generated by a Gaussian distribution. We have conducted several experiments on very high-resolution satellite imagery, representing four unique geographic regions across the world. Our method showed consistent improvement in accurately identifying informal settlements.

Graphs play a role in many semi-supervised learning algorithms, where unlabeled samples are used to find useful structural properties in the data. Dimensionality reduction and regularization based on preserving smoothness over a graph are common in these settings, and they perform particularly well if proximity in the original feature space closely reflects similarity in the classification problem of interest. However, many real-world problem spaces are overwhelmed by noise in the form of features that have no useful relevance to the concept that is being learned. This leads to a lack of robustness in these methods that limits their applicability to new domains. We present a graph-construction method that uses a collection of task-specific random subspaces to promote smoothness with respect to the problem of interest. Application of this method in a graph-based semi-supervised setting demonstrates improvements in both the effectiveness and robustness of the learning algorithms in noisy problem domains.

As a result of increasing geospatial image libraries, many algorithms are being developed to automatically extract and classify regions of interest from these images. However, limited work has been done to compare, validate and verify these algorithms due to the lack of datasets with high accuracy ground truth annotations. In this paper, we present an approach to generate a large number of synthetic images accompanied by perfect ground truth annotation via learning scene statistics from few training images through Maximum Entropy (ME) modeling. The ME model [1,2] embeds a Stochastic Context Free Grammar (SCFG) to model object attribute variations with Markov Random Fields (MRF) with the final goal of modeling contextual relations between objects. Using this model, 3D scenes are generated by configuring a 3D object model to obey the learned scene statistics. Finally, these plausible 3D scenes are captured by ray tracing software to produce synthetic images with the corresponding ground truth annotations that are useful for evaluating the performance of a variety of image analysis algorithms.

The design of benchmark imagery for validation of image an notation algorithms is considered. Emphasis is placed on imagery that contains industrial facilities, such as chemical re fineries. An application-level facility ontology is used as a means to define salient objects in the benchmark imagery. Instrinsic and extrinsic scene factors important for comprehensive validation are listed, and variability in the benchmarks discussed. Finally, the pros and cons of three forms of bench mark imagery: real, composite and synthetic, are delineated.

Accurate damage assessment due to major natural and anthropogenic disasters is becoming critical due to increasing human and economic losses. This increase in loss of life and severe damages can be attributed to the growing population, and human migration and settlements in disaster prone regions of the world. Rapid damage assessment and dissemination of accurate information is critical for creating an effective emergency response. Remote sensing and geographic information systems (GIS) based techniques and tools are playing an important function in disaster damage assessment and reporting activities. In this review, we will look into the current state of art, in damage assessment using remote sensing and GIS based techniques.

Online time series change detection is a critical component of many monitoring systems, such as space and air-borne remote sensing instruments, cardiac monitors, and network traffic profilers, which continuously analyze observations recorded by sensors. Data collected by such sensors typically has a periodic component. Most existing time series change detection methods are not directly applicable to handle such data, either because they are not designed to handle periodic time series or because they cannot operate in an online mode. We propose an online change detection algorithm which can handle periodic time series. The algorithm uses a Gaussian process based non-parametric time series prediction model and monitors the difference between the predictions and actual observations within a statistical control chart framework to identify changes. A key challenge in using Gaussian process in an online mode is the need to solve a large system of equations involving the associated covariance matrix which grows with every time step. The proposed algorithm exploits the special structure of the covariance matrix and can analyze a time series of length T in O(T2) time while maintaining a O(T) memory footprint, compared to O(T4) time and O(T2) memory requirement of standard matrix manipulation methods. We experimentally demonstrate the superiority of the proposed algorithm over several existing time series change detection algorithms on a set of synthetic and real time series. Finally, we illustrate the effectiveness of the proposed algorithm for identifying land use land cover changes using Normalized Difference Vegetation Index (NDVI) data collected for an agricultural region in Iowa state, USA. Our algorithm is able to detect different types of changes in a NDVI validation data set (with ≈ 80% accuracy) which occur due to crop type changes as well as natural disasters.

Identification and automatic labeling of facilities in high-resolution satellite images is a challenging task as the current thematic classification schemes and the low-level image features are not good enough to capture complex objects and their spatial relationships. In this paper we present a novel algorithm framework for automated semantic labeling of large image collections. The framework consists of various segmentation, feature extraction, vector quantization, and Latent Dirichlet Allocation modules. Initial experimental results show promise as well as the challenges in semantic classification technology development for nuclear proliferation monitoring.

With increasing understanding and availability of nuclear technologies, and increasing persuasion of nuclear technologies by several new countries, it is increasingly becoming important to monitor the nuclear proliferation activities. There is a great need for developing technologies to automatically or semi-automatically detect nuclear proliferation activities using remote sensing. Images acquired from earth observation satellites is an important source of information in detecting proliferation activities. High-resolution remote sensing images are highly useful in verifying the correctness, as well as completeness of any nuclear program. DOE national laboratories are interested in detecting nuclear proliferation by developing advanced geospatial image mining algorithms. In this paper we describe the current understanding of geospatial image mining techniques and enumerate key gaps and identify future research needs in the context of nuclear proliferation.

We demonstrate a Google Earth based novel visualization system for continuous monitoring of biomass at regional and global scales. This system is integrated with a back-end spatiotemporal data mining system that continuously detects changes using high temporal resolution MODIS images. In addition to the visualization, we demonstrate novel query features of the system that provides insights into the current conditions of the landscape.

Intelligent image information mining for thematic pattern extraction is a complex task. Ever increasing spatial, spectral, and temporal resolution poses several challenges to the geographic knowledge discovery community. Although the improvements in sensor technology and data collection methods may lead to improved geoinformation generation, it also places several constraints on data mining techniques. Moreover thematic classes are spectrally overlapping, that is, many thematic classes can not be separated by spectral features alone. In recent years we have developed several innovative machine learning approaches to address these problems. The resulting software system, called *Miner, was tested on several real world multisource spatiotemporal datasets. Experimental evaluation showed improved accuracy over conventional data mining approaches. In addition, we integrated *Miner with another popular open source machine learning system called Weka. In this demo we show the utility of *Miner for thematic information extraction from multisource spatiotemporal data (remote sensing images and ancillary geospatial databases).

In many practical situations it is not feasible to collect labeled samples for all available classes in a domain. Especially in supervised classification of remotely sensed images it is impossible to collect ground truth information over large geographic regions for all thematic classes. As a result often analysts collect labels for aggregate classes. In this paper we present a novel learning scheme that automatically learns sub-classes from the user given aggregate classes. We model each aggregate class as finite Gaussian mixture instead of classical assumption of unimodal Gaussian per class. The number of components in each finite Gaussian mixture are automatically estimated. Experimental results on real remotely sensed image classification showed not only improved accuracy in aggregate class classification but the proposed method also recognized sub-classes.

In many practical situations thematic classes can not be discriminated by spectral measurements alone. Often one needs additional features such as population density, road density, wetlands, elevation, soil types, etc. which are discrete attributes. On the other hand remote sensing image features are continuous attributes. Finding a suitable statistical model and estimation of parameters is a challenging task in multisource (e.g., discrete and continuous attributes) data classification. In this paper we present a semi-supervised learning method by assuming that the samples were generated by a mixture model, where each component could be either a continuous or discrete distribution. Overall classification accuracy of the proposed method is improved by 12% in our initial experiments.

New approaches are needed to extract useful patterns from increasingly large multi-spectral remote sensing image databases in order to understand global climatic changes, vegetation dynamics, ocean processes, etc. Supervised learning, which is often used in land cover (thematic) classification of remote sensing imagery, requires large amounts of accurate training data. However, in many situations it is very difficult to collect labels for all training samples. In this paper we explore methods that utilize unlabeled samples in supervised learning for thematic information extraction from remote sensing imagery. Our objectives are to understand the impact of parameter estimation with small learning samples on classification accuracy, and to augment the parameter estimation with unlabeled training samples to improve land cover predictions. We have developed a semi-supervised learning method based on the expectation-maximization (EM) algorithm, and maximum likelihood and maximum a posteriori classifiers. This scheme utilizes a small set of labeled and a large number of unlabeled training samples. We have conducted several experiments on multi-spectral images to understand the impact of unlabeled samples on the classification performance. Our study shows that though in general classification accuracy improves with the addition of unlabeled training samples, it is not guaranteed to get consistently higher accuracies unless sufficient care is exercised when designing a semi-supervised classifier

Modeling spatial context (e.g., autocorrelation) is a key challenge in classification problems that arise in geospatial domains. Markov random fields (MRF) is a popular model for incorporating spatial context into image segmentation and land-use classification problems. The spatial autoregression (SAR) model, which is an extension of the classical regression model for incorporating spatial dependence, is popular for prediction and classification of spatial data in regional economics, natural resources, and ecological studies. There is little literature comparing these alternative approaches to facilitate the exchange of ideas. We argue that the SAR model makes more restrictive assumptions about the distribution of feature values and class boundaries than MRF. The relationship between SAR and MRF is analogous to the relationship between regression and Bayesian classifiers. This paper provides comparisons between the two models using a probabilistic and an experimental framework.

The integration of disparate heterogeneous spatial databases for extending queries is a challenging task. The authors present a novel framework, based on a k-nearest neighbor (kNN) algorithm, for integrating remote sensing imagery with Forest Inventory Analysis (FIA) sample point/plot data managed in a relational database system. We then demonstrate how queries to this system may be extended over any arbitrary region of interest in a Web based geographical information system. To build the integrated database, spectral signatures are collected at FIA plot locations from the Landsat TM image. A plot-id image is produced by assigning each pixel to the closest FIA plot in multi-dimensional spectral space. The resulting image provides an interface to the Forest Inventory Analysis Data-Base (FIADB) and allows generalizations of the estimates for any user defined query window or region of interest (ROI). This methodology, along with geostatistical analysis, is integrated into a client/server Web based geographical information system, which provides Internet users with an easy to use query interface for the FIADB and spatial databases.