While the availability of multispectral imagery (MSI) is increasing, there is still much to be learned regarding the spectral sensitivity of deep neural networks (DNNs). This work presents and analyzes DNNs trained to perform object detection in high-resolution multispectral, satellite imagery. These DNNs are first trained and evaluated on variations of the xView dataset with varied bit-depths and spectral band inclusion. Inference is also performed on test imagery with different bit-depths or with missing spectral information in order to analyze their importance for object detection. Initial experiments show an MSI-trained DNN had an average optimal F1-score 3.7 points higher than the best performing RGB network. Additional experiments suggest significant DNN sensitivity to changes in bit-depth and an even greater sensitivity to the loss of any given spectral band. The results given here highlight the need for future research regarding spectral biases of DNNs for object detection in high-resolution MSI.

Land valuations are used by buyers, sellers, regulators, and authorities to assess fair value. Urbanization demands a modern and efficient land valuation system. An alternative method that integrates geographic information systems and machine learning can produce more reliable, repeatable, and accurate valuations. The land’s value can vary significantly based on the parcel’s neighborhood and context. Geospatial analytics, remote sensing and vector information were integrated to predict land values in Springfield, Missouri, USA. A point geodatabase was assembled using calculated proximities, accessibility, and various index measures. Supervised machine learning models were trained using government-provided appraised land values as ground truth. The database approach integrated spatial context and socioeconomic data. Slight differences in performance between random forest (98%, 0.13) and gradient boosting (98%, 0.15) algorithms were found (Adj. R2, STD).

Deforestation poses a critical environmental challenge with far-reaching impacts on climate change, biodiversity, and local communities. As such, detecting and monitoring deforestation are crucial, and recent advancements in deep learning and remote sensing technologies offer a promising solution to this challenge. In this study, we adapt ChangeFormer, a transformer-based framework, to detect deforestation in the Brazilian Amazon, employing the attention mechanism to analyze spatial and temporal patterns in bi-temporal satellite images. To assess the model’s effectiveness, we employed a robust approach to create a deforestation detection dataset, utilizing Sentinel-2 imagery from select conservation areas in the Brazilian Amazon throughout 2020 and 2021. Our dataset comprises 7,734 pairs of bi-temporal image chips with a resolution of 256×256 pixels and 1,406 pairs of image chips with a resolution of 512×512 pixels. The model achieved an overall accuracy of 93% with corresponding F1 score of 90% and IoU score of 82%. These results demonstrate the potential of transformer-based networks for accurate and efficient deforestation detection.

As the demand for real-time signal processing increases in various fields, such as healthcare, artificial intelligence, machine learning, and scientific research, there is a need for more efficient methods to analyze large amounts of data. To address this challenge and explore the opportunities to accelerate different signal processing algorithms, this paper proposes the integration of graphics processing units (GPUs) with database management systems (DBMS) using the PostgreSQL server programming interface (SPI). The performance of the proposed method is evaluated by comparing central processing unit (CPU) and GPU approaches for feature extraction using a data processing pipeline for heart rate estimation from hydraulic bed sensor data. Furthermore, the paper analyzes timing metrics, usability, adaptability, and discusses precision differences between CPU and GPU code by performing different thread and block configurations.

Large numbers of labeled data are necessary for the success of modern deep-learning techniques. Despite a large amount of satellite image data now available, their ground truth labels are inadequate due to the complexity of the real world. It is true for many remote sensing tasks including scene classification, target classification, and target detection. This study explores and evaluates a state-of-the-art pipeline that combines transfer learning and meta-learning methods with voluminous external data for few-shot remote sensing scene classification. The experimental findings demonstrate that using this pipeline, both in-domain and out-of-domain data can lead to equivalent performance as the base data during training. Additionally, the study explores the impact of various N-way-K-shot tasks in the meta-training stage and finds that the model trained with 5-way-5-shot tasks achieves the highest level of performance.

Deep neural network (DNN) algorithms have increased dramatically across a wide variety of remote sensing applications. However, as computer vision (CV) researchers have utilized the power of DNN, additional shortcomings have become more clear, including the reliance of DNNs on texture information rather than shape information to perform feature extraction for CV tasks. This lack of shape can negatively impact many remote sensing applications, where shape information can provide the key discriminatory features in object detection. To improve the utilization of shape information in DNNs, a novel model architecture was developed that incorporated shape information extracted using the Differential Morphological Profile (DMP), which was directly embedded in the DNN. While this network showed the ability to outperform traditional DNNs in CV tasks, there existed several instances in which the tradeoff of spectral and shape information proved ineffective. In this study, we present a Hybrid DMP-enabled network that utilizes both the spectral information of traditional DNNs and the shape information extracted by DMP. Our results show that this network is capable of outperforming competing methods in overhead remote sensing object detection.

Earth Observation (EO) data have become abundantly available and at low prices or for free, opening many possibilities to tackle problems such as the mapping of burned areas after wildfires have been extinguished. This task can be completed using deep learning semantic segmentation techniques. In this paper, we first present our methodology for adapting burn area polygons into semantic segmentation training data for deep neural models. Then, we evaluate four deep semantic segmentation models, U-Net, U-Net++, DeepLabV3, and DeepLabV3+ on Sentinel-2 satellite imagery data obtained from the Copernicus program over Canada in 2019. Our results show that DeepLab models outperformed U-Net models with the DeepLabV3 model achieving the best recall of 83.78%, while the DeepLabV3+ achieved the best precision of 84.35%. EO data allows us a faster and more accurate assessment of burned areas, which can accelerate the restoration planning and processing of insurance claims.

or respiratory illnesses due to heart-related issues are often misdiagnosed, under-diagnosed or ignored at early stages. Continuous health monitoring using ambient sensors has the potential to ameliorate this problem for older adults at aging-in-place facilities. In this paper, we leverage continuous respiratory health data collected by using ambient hydraulic bed sensors installed in the apartments of older adults in aging-in-place Americare facilities to find data-adaptive indicators related to shortness of breath. We used unlabeled data collected unobtrusively over the span of three years from a COPD-diagnosed individual and used data mining to label the data. These labeled data are then used to train a predictive model to make future predictions in older adults related to shortness of breath abnormality. To pick the continuous changes in respiratory health we make predictions for shorter time windows (60-s). Hence, to summarize each day's predictions we propose an abnormal breathing index (ABI) in this paper. To showcase the trajectory of the shortness of breath abnormality over time (in terms of days), we also propose trend analysis on the ABI quarterly and incrementally. We have evaluated six individual cases retrospectively to highlight the potential and use cases of our approach.

With the increasing interconnection of smart sensors in long-term care facilities, the amount of data available for multi-modal Big Data analytics is advantageous. Using raw smart sensor data, researchers can derive and extract physiological features useful for health monitoring. Nonetheless, with the immense amount of smart sensor data, the ability to utilize these data for multi-modal analytics as the data grows presents a great challenge for researchers and long-term care facilities. This paper proposes a database design system for multi-modal derived time-series featured data (respiration and restlessness) by using techniques such as hierarchical time-indexed databases and dense numerical array storage. We present evaluations and findings for our proposed database system design for multi-modal time-series feature data to assess the various performance characteristics in data access time, storage, and usability; demonstrating an extremely scalable design and simple integration with existing analytic tools via SQL interfaces. Furthermore, we introduce a data-processing pipeline enabling Big Data analytics for multi-modal time-series feature data.

The spatial configuration of objects in a scene is important to many applications. In particular, 3D environments constructed from point cloud observations are often used for navigation and planning with real-time requirements. In these settings, the ability to recognize and distinguish one set of objects from another may depend largely on how they are positioned with respect to each other. In this article, we explore two different approaches for describing the relative spatial relationship between two objects represented as 3D points: the histogram of forces, and a method using bounding boxes and fuzzy numbers. We use 2D axis-aligned projections of the objects to facilitate the computation of force histograms, and compare this approach to the bounding box and fuzzy number method. Our experiments are performed on the NPM3D dataset, consisting of hand-labeled point cloud objects in an outdoor street-level environment. The results highlight the strengths and weaknesses of each approach and we discuss the most appropriate applications for both.

Modern satellite sensors can capture reflected optical energy from wavelengths well outside of the range of human perception. Indeed, it is well known that electromagnetic energy from the “non-visible” spectrum can be used to identify materials in geological, oceanographic, and agricultural contexts. What is less understood, however, is if and how this additional spectral information can be leveraged to aid in difficult computer vision tasks, e.g., classification or detection of man-made objects. Thus, we present the results from a series of experiments that evaluate the benefits that Deep Convolutional Neural Networks (DCNN) can garner through the inclusion of more spectral information. For training and testing these networks in image classification, we extract image tiles from the xView multi-spectral dataset. These images contain eight channels of spectral data including five wavelengths bands beyond the three bands traditionally used in computer vision researcher (red, green, and blue). In this work, we report the results of experiments that use an 80–20 train-test split with four DCNN architectures on full 8-band Multispectral Imagery (MSI) and various subsets of this eight banded imagery. The results show that networks trained on MSI have an average testing F1-score around 1.0 point higher than RGB networks trained with the same methods.

Road network understanding is an important component of determining where a vehicle can safely maneuver in an area of interest, especially in compromised environment scenarios such as disaster response. Low-altitude unmanned aircraft systems and semantic segmentation via deep neural networks provide an efficient solution to denoting the locations of roads in potentially large swaths of imagery. Unmanned aerial vehicles such as drones can be flown over an area of interest to quickly capture high quality overhead images in multiple modalities such as visible and infrared; and can provide updated imagery as needed, which is important for rapidly changing situations. Meanwhile, deep neural networks can automatically segment the roads in the incoming imagery. In this work we evaluate two semantic segmentation deep neural network architectures, U-Net and DeepLabV3+, and their ability to segment roads in low-altitude UAS imagery in both the visible and infrared modalities. We find that segmenting roads in the infrared imagery is more difficult for the models, and our results indicate that DeepLabV3+ may have the ability to better generalize to unseen data than U-Net when data augmentation is introduced. Our evaluations show an mIoU of 87.33 and 82.47 on the test sets for the visible and infrared domains, respectively, using DeepLabV3+.

Recently, computer vision tasks such as classification and object detection have been dominated by deep neural net-work (DNN) approaches. As DNN methodologies have matured, researchers have found that some of the most common DNN techniques result in models that are highly dependent upon the textures and colors of the imagery, rather than the shape, leading to suboptimal network performance. This problem can be especially problematic in the remote sensing domain, where the discrimination of objects for classification or detection may rely heavily on their shape. To combat this lack of shape bias in DNNs, a network was developed to integrate the Differential Morphological Profile (DMP), an image processing technique for shape extraction, with standard convolutional DNNs for performing computer vision tasks on High Resolution Remote Sensing Imagery (HR-RSI). Previously, this network, known as DMPNet, has been applied to both classification and object detection in HR-RSI with high levels of success. However, the hyper-parametric nature of DMPNet structure required researchers to carefully select the parameters of shape extraction, a choice that could greatly help or hinder DMPNet performance. In this study, we utilize a evolutionary computation algorithm (ECA) to learn the parameters of shape extraction from the data presented to the DMPNet for object detection. Our results show that our DMP-enabled detection models perform better object detection in HR-RSI using an ECA to learn shape extraction parameters than manually selected parameters on the same dataset.

Three dimensional point cloud data sets are easy to acquire and manipulate, but are often too large to process directly for embedded real-time applications. The spatial information in a point cloud can be represented in a variety of reduced forms, such as voxel grids, Gaussian mixture models, or spatial semantic structures. In this article, we show how a segmented point cloud can be represented as a spatial relationship graph using bounding boxes and triangular fuzzy numbers. This model is a lightweight encoding of the relative distance and direction between objects, and can be used to describe and query for particular spatial configurations using linguistic terms in a multicriteria framework. We show how this approach can be applied on a hand-segmented subset of the NPM3D data set with several illustrative examples. The work herein has useful applications in many applied domains, such as human-robot interaction with unmanned aerial systems.

A prevalent challenge in underdeveloped countries is the mapping and accounting of certain sub-populations for public health matters. This is especially true in sub-Saharan African countries that lack specific resources and technologies. Often, international non-profit groups provide targeted support for populations' health on specific matters, such as clean water initiatives and basic health clinics. This is especially critical in various ethnic populations that are not well integrated into a country's existing public health services. In countries such as Tanzania, one such ethnic population is the Maasai people. In this paper, we demonstrate how machine learning algorithms, combined with remote sensing data, can aid non-profit health delivery organizations to map Maasailand boma homesteads in northern Tanzania. Our study area encompasses over 3900 square kilometers of Massailand, where we have manually identified 635 bomas. We have evaluated four different deep neural network architectures for their ability to recognize boma, ResNet50, Xception, Inception-ResNet-V2, and ProxylessNAS, with cross-validation F1 scores of 95.32%, 95.45%, 96.78%, and 97.28%, respectively.

Deep learning has proven to be an immensely powerful tool within the remote sensing space, with capabilities to perform tasks like classification, object detection, and segmentation in a wide range of modalities and spatial resolutions. The deep neural networks (DNN) extract visual features using numerous techniques, including convolutional and pooling layers, residual modules, inception modules, neural architecture search, and attention networks. Researchers have increasingly found that DNN are biased towards texture, and that this bias is a deficiency that when corrected, can boost classification and detection performance of deep learners. A morphology-based network utilizing the differential morphological profile (DMP) as a non-parametric feature extraction layer, known as DMPNet, was proposed with promising results when compared to VGG16. In this work, the original DMPNet is expanded with increased profile depth, as well as alternate convolutional phases, such as ResNet-18 and MobileNet. The resulting architecture shows an ability to increase shape information in the network, and with it, generalizability on high resolution remote sensing imagery.

While most deep learning architectures are built on convolution, alternative foundations such as morphology are being explored for purposes such as interpretability and its connection to the analysis and processing of geometric structures. The morphological hit-or-miss operation has the advantage that it considers both foreground information and background information when evaluating the target shape in an image. In this article, we identify limitations in the existing hit-or-miss neural definitions and formulate an optimization problem to learn the transform relative to deeper architectures. To this end, we model the semantically important condition that the intersection of the hit and miss structuring elements (SEs) should be empty and present a way to express Don't Care (DNC), which is important for denoting regions of an SE that are not relevant to detecting a target pattern. Our analysis shows that convolution, in fact, acts like a hit-to-miss transform through semantic interpretation of its filter differences. On these premises, we introduce an extension that outperforms conventional convolution on benchmark data. Quantitative experiments are provided on synthetic and benchmark data, showing that the direct encoding hit-or-miss transform provides better interpretability on learned shapes consistent with objects, whereas our morphologically inspired generalized convolution yields higher classification accuracy. Finally, qualitative hit and miss filter visualizations are provided relative to single morphological layer.

We are currently experiencing a deluge of high-resolution electroptical (HR-EO) remote sensing images which can be leveraged for a diverse set of applications, ranging from environmental monitoring to defense and security applications. One of the most challenging application domains is the use of machine learning techniques, such as computer vision, for the enhancement and automation of geospatial big data analytics. In this work, we present techniques for extending deep convolutional neural networks (DCNN) from 3-band color imagery to 4-band and 8-band multispectral remote sensing imagery. Performance comparisons are conducted between DCNN for 3-, 4-, and 8-band imagery data using the Functional Map of the World dataset. In particular, we investigate five distinct DCNN architectures for classification, and show that utilizing more input channels of the imagery typically has a positive impact on classification metrics such as F1-score and raw accuracy. Herein, we evaluated two methods for initializing the DCNN convolutional filters with 4-or 8-band inputs via transfer learning from networks trained on 3-band (RGB) images and show these methods are superior to random initialization. Our findings indicate that additional spectral bands have varied benefits, depending on the image class, where challenging classes saw minimal improvements. We detail these findings along with insights into the implications for multispectral DCNN in particular geospatial big data analytics use-cases.

In the last decade, several high resolution remote sensing benchmark datasets have been developed and publicly released. These datasets, while diverse in design, lack the required intra-class variation for high-performing, machine-assisted visual analytics. More specifically, the disparate datasets are suitable for small, closed system evaluation; however they are not well suited for training of computer vision models that are robust in real-world, non-closed environments encountered in true remote sensing applications. To that end, a benchmark meta-dataset (MDS) was developed to facilitate the training of models for machine-assisted visual analytics. Four existing benchmark datasets were combined to build the original MDS, which excelled for training models for both classification and broad area search applications. In this work, we evaluate an enhanced version of the MDS, MDSv2, by integrating co-occurring classes of two additional recently released, publicly available challenge datasets: xView and Functional Map of the World (FMoW). The MDSv2 has 33 classes with 87,470 total samples. We investigate the utility of three neural architecture search (NAS) deep learning architectures on the MDSv2 for both classification and machine-assisted visual analytics. The NAS models trained with the MDSv2 are able to achieve an average F1 of 98.01% and a powerful 0.934 scanning precision.

Artificial neural networks are a dominant force in our modern era of data-driven artificial intelligence. The adaptive neuro fuzzy inference system (ANFIS) is a neural network based on fuzzy logic versus a more traditional premise like convolution. Advantages of ANFIS include the ability to encode and potentially understand machine learned neural information in the pursuit of explainable, interpretable, and ultimately trustworthy artificial intelligence. However, real-world data is almost always imperfect, e.g., incomplete or noisy, and ANFIS is not naturally robust. Specifically, ANFIS is susceptible to over inflated uncertainty, poor antecedent (fuzzy set) data alignment, degenerate optimization conditions, and hard to interpret logic, to name a few factors. Herein, we explore the use of possibilistic clustering to identify outliers, specifically typicality degrees, to increase the robustness of ANFIS; or any fuzzy logic neuron/network. Experiments are presented that demonstrate the need and quality of the proposed solutions in the pursuit of robust interpretable machine learned neuro fuzzy logic solutions.

Deep convolutional neural networks (DCNN) have been the dominant methodology in the field of computer vision over the last decade, using various architectural organizations of successive convolutional layers to extract and assemble low level image features into visual component detectors. One of the tradeoffs that have been made as the community has migrated to deep neural models is the loss of explainability and understanding of which salient visual components are being recognized by a model for a particular task. However, there exists a significant heritage in the remote sensing community that has developed advanced algorithms to analyze the signal and structural characteristics of anthropogenic features. One such approach is the use of morphological image processing techniques to extract objects from imagery and aid in the structural analysis of shapes. In particular, the differential morphological profile (DMP) has had great success extracting object shapes, while naturally grouping the extracted shapes into scale ranges. In this research, we present a novel architecture that integrates an explicit (definable and explainable) scaled object extraction into the network architecture, allowing shallower convolutional layers and lower complexity neural models. The architecture is evaluated on a challenging remote sensing dataset of object classes, providing insights to this approach and illuminating future directions of integrating morphology into neural architectures for enhanced explainability.

In this letter, we propose a chip-based technique for large-scale, automatic land cover clustering of high-resolution remote sensing imagery (HR-RSI) using deep visual features from a deep convolutional neural network (DCNN) along with the fuzzy c-means algorithm. The proposed method, unlike traditional methods, facilitates utilizing transfer learning techniques for deep neural models that are fined-tuned to satellite imagery for feature extraction. Then, a large conterminous region of imagery is acquired from HR-RSI data providers and scanned to extract visual features utilizing the transfer learned model. Broad-area thematic and contextual understanding of the geographic land cover is efficiently achieved using feature reduction, chip-based clustering analysis, and geospatial rendering of the clusters. We explore a variety of fuzzy clusterings and their resulting utility for spatial analysis. The spatial densities of the clusters, numerical analysis, and geographic aggregations show that our proposed approach is effective in examining the land cover of the earth.

Information fusion is an essential part of numerous engineering systems and biological functions, e.g., human cognition. Fusion occurs at many levels, ranging from the low-level combination of signals to the high-level aggregation of heterogeneous decision-making processes. While the last decade has witnessed an explosion of research in deep learning, fusion in neural networks has not observed the same revolution. Specifically, most neural fusion approaches are ad hoc, are not understood, are distributed versus localized, and/or explainability is low (if present at all). Herein, we prove that the fuzzy Choquet integral (ChI), a powerful nonlinear aggregation function, can be represented as a multilayer network, referred to hereafter as ChIMP. We also put forth an improved ChIMP (iChIMP) that leads to a stochastic-gradient-descent-based optimization in light of the exponential number of ChI inequality constraints. An additional benefit of ChIMP/iChIMP is that it enables explainable artificial intelligence (XAI). Synthetic validation experiments are provided, and iChIMP is applied to the fusion of a set of heterogeneous architecture deep models in remote sensing. We show an improvement in model accuracy, and our previously established XAI indices shed light on the quality of our data, model, and its decisions.

Here we demonstrate how Deep Neural Network (DNN) detections of multiple constitutive or component objects that are part of a larger, more complex, and encompassing feature can be spatially fused to improve the detection performance of a larger complex feature. A wide variety of experiments were conducted using the public domain xView dataset to develop and evaluate multiple fusion strategies to improve the detection of Construction Sites using DNN detections of constitutive/component objects commonly associated with construction activity, e.g. cement mixers, dump trucks, etc. The results demonstrate that spatial fusion of multi-scale component object DNN detections can reduce the total detection error rate of Construction Sites by ~30-40%. The best results were obtained when local spatial clustering was used to reduce noise in component vehicle object detections generated by scanning candidate Construction Site locations. This multi-scale spatial fusion approach can be easily extended to improve detection performance in a wide variety of other challenging feature/object search and detection problems in large-scale remote sensing image datasets.

Object detection is a challenging but important task for computer vision, and this is especially true in the remote sensing domain where data collections may bring billions of pixels in a single image. There are many methods for object detection, but in recent years the You Only Look Once (YOLO) algorithm has become a leading technique, gaining popularity due to its ability to perform real-time object detection. While YOLO and its successors have shown excellent results in realtime detection, there are many object detection tasks that require better precision, and do not require real-time detection. In this paper, YOLOv3 is compared to other deep neural networks (DNN) for detecting Vehicle Groups in very high resolution remote sensing imagery (VHR-RSI). A unique centerpoint-based dataset is developed by leveraging a novel data framework and combining quality assured chips with regions of interest in the XView Challenge Dataset. This dataset is then used to train state of the art models including two Neural Architecture Search (NAS) variant DNN for object detection. Additionally, a blind test set is developed to further compare our methods with the YOLOv3 algorithm. The results shows that our method detects vehicle groups with a lower false positive rate (FPR) and better true positive rate (TPR) than state-of-the-art YOLOv3 models for the blind test set; achieving a reduction in error rate of 26.70% over YOLOv3 in F1 Score on the blind test set.

Object detection and localization within high-resolution remote sensing imagery (HR-RSI) is a challenging task for a variety of reasons, such as the complexity and clutter of the image scene and the compactness of the intermixed object classes. Even the most comprehensive training datasets cannot adequately account for the rich diversity and complexity of anthropogenic objects and their contextual settings in large-scale HR-RSI collections. Recent approaches using deep learning techniques include bounding box approaches (e.g., YOLO), object nomination then recognition (e.g., R-CNN), and post-detection object localization of deep neural network detections. Herein, we propose a novel technique that leverages heterogeneous superpixels and deep neural feature extraction to classify the superpixel segmentation through relational analysis. In this preliminary research, we demonstrate the validity of this approach for object detection and localization, as well as its suitability for identifying the irregular shapes of objects (as opposed to a bounding box). Experiments are performed using a sub-set of the xView benchmark dataset with a goal of spearheading future techniques in cluttered scene object recognition that allows deep feature extractors to have more focus on the target objects instead of the surrounding area or nearby object pixels.

The Choquet integral (ChI) is a proven tool for information aggregation. In prior work, we showed that learning a ChI from data results in missing variables. Herein, we explore two ways to transfer a known ChI from a source domain to a new under sampled target domain. The first method is based on regularization and it listens to the full source domain ChI. The second method optimizes what we can observe (target domain supported variables) and missing variables are the only thing migrated from the source domain. Synthetic experiments, aka we know the truth, are used to show the behavior of these methods with respect to transfering between ChIs.

Herein, a generalization of the ordered weighted average (OWA) is put forth relative to pattern recognition. The resultant linear order statistic neuron (LOSN) is unique in that it bridges fuzzy sets, specifically fuzzy data/information aggregation, with neural networks. This article discusses the gradient descent-based optimization and geometric interpretation of the LOSN. An advantage is that the LOSN is an efficient shared weight encoding of N! perceptrons, relative to N inputs. Open source codes are provided to facilitate reproducible research. Experiments are conducted to both validate the method and show its non-linear geometric expression.

Here we demonstrate how Deep Neural Network (DNN) detections of multiple constitutive or component objects that are part of a larger, more complex, and encompassing feature can be spatially fused to improve the search, detection, and retrieval (ranking) of the larger complex feature. Scores computed from a spatial clustering algorithm are normalized to a reference space so that they are independent of image resolution and DNN input chip size. DNN detections from multiple component objects can then be fused with or without human-expert provided weights to improve the retrieval (ranking) of DNN detections of a larger complex feature. We demonstrate the utility of this approach for broad area search and detection of Surface-to-Air Missile (SAM) sites that have a very low occurrence rate (only 16 sites) over a ~90,000 km2study area in SE China. Our spatial fusion approach can be easily extended to a wide variety of other challenging object search and detection problems in large-scale remote sensing image datasets.

Recently, many high-resolution remote sensing imagery (HR-RSI) datasets have been released that have diverse characteristics, such as high inter-class and low intra-class variation. Additionally, a benchmark meta-dataset (MDS) was created by agglomerating object classes from multiple HR-RSI datasets. Previous work has shown that deep convolutional neural networks (DCNN) trained on the MDS perform on par with DCNN trained on constituent benchmark datasets in cross-validation experiments. Here we train a model ensemble on four datasets and compare it with a single robust DCNN trained on the MDS. The goal is to better understand under what conditions an ensemble of models, each trained with distinct datasets, is advantaged or disadvantaged compared to a single model trained with the agglomerated MDS for classification performance.

The Internet of Things (IoT) continues to expand; as daily new smart-devices are connected to Internet and adding to a deluge of data created by our society. Compounding the challenges is that this data is very heterogeneous, including device or human activity trace data, structured data, sensor information, and media data. The computing needs for the IoT continue to rise in terms of scalable compute power, storage, and complex data processing pipelines to accommodate these diverse sources of data. For this reason, it becomes essential to develop a flexible framework that is able to efficiently manage the IoT data in a real-time and scalable approach. In this paper, we propose a novel framework to handle IoT data. Our framework is dynamically extensible, lightweight, resources efficient, and has the ability to handle stream data as well as batch data. We leverage autonomous agents along with the publish-subscribe pattern to achieve a run-time extensible, event-driven, and high-performance computational architecture. Additionally, we have incorporated localized and centralized databases into the framework to support structured and unstructured data for compute processing and analytical tasks. We have implemented the proposed framework and evaluated its performance using a visual object-detection case study on both a local cluster and within cloud-computing infrastructure. Our analysis shows that this framework utilizes the CPU, memory, and network resources efficiently. Additionally, the framework can scale horizontally as adding more processing nodes reduces the time and increases the goodput.

To date, numerous ways have been created to learn a fusion solution from data. However, a gap exists in terms of understanding the quality of what was learned and how trustworthy the fusion is for future-i.e., new-data. In part, the current paper is driven by the demand for so-called explainable AI (XAI). Herein, we discuss methods for XAI of the Choquet integral (ChI), a parametric nonlinear aggregation function. Specifically, we review existing indices, and we introduce new data-centric XAI tools. These various XAI-ChI methods are explored in the context of fusing a set of heterogeneous deep convolutional neural networks for remote sensing.

Here we present techniques and algorithms to apply trained deep convolutional neural networks (DCNN) to high-resolution remote sensing imagery datasets covering large areas of the Earth. First, trained DCNN are used to process broad swaths of imagery in an area of interest (AOI) to produce a classification vector response field (CVRF). The CVRF is then aggregated using mode-seeking algorithms to detect potential objects of interest within the AOI. Our research explores the challenges and opportunities of transitioning DCNN out of the training-validation laboratory setting and into the real-world application domain. We show a scalable approach to leverage state-of-the-art DCNN for broad area automated search, detection, and annotation of objects such as tennis courts, storage tanks, runways, and airplanes.

Accurate land cover classification and detection of objects in high-resolution electro-optical remote sensing imagery (RSI) have long been a challenging task. Recently, important new benchmark data sets have been released which are suitable for land cover classification and object detection research. Here, we present state-of-the-art results for four benchmark data sets using a variety of deep convolutional neural networks (DCNN) and multiple network fusion techniques. We achieve 99.70%, 99.66%, 97.74%, and 97.30% classification accuracies on the PatternNet, RSI-CB256, aerial image, and RESISC-45 data sets, respectively, using the Choquet integral with a novel data-driven optimization method presented in this letter. The relative reduction in classification errors achieved by this data driven optimization is 25%-45% compared with the single best DCNN results.

Data retrieval serves a critical role in the development of multimedia applications. However, due to the exponential growth of multimedia data, high-speed and efficient indexing is becoming more and more difficult than ever. In this paper, we propose a novel approach to speed up the retrieval process by adopting a distributed computing paradigm through the Apache Spark framework. Utilizing search trees in a Big Data ecosystem leads to fast and cost-effective media database retrievals by caching indexing structures into memory and aggregating ranked results with flexibilities for users to specify the importance of search cues. We conducted computational experiments on large-scaled vector files for remote sensing image database and synthesized pollen image database to demonstrate the effectiveness and scalability of our system with reasonably high accuracy.

Deep convolutional neural networks (DCNNs) have recently emerged as a dominant paradigm for machine learning in a variety of domains. However, acquiring a suitably large data set for training DCNN is often a significant challenge. This is a major issue in the remote sensing domain, where we have extremely large collections of satellite and aerial imagery, but lack the rich label information that is often readily available for other image modalities. In this letter, we investigate the use of DCNN for land-cover classification in high-resolution remote sensing imagery. To overcome the lack of massive labeled remote-sensing image data sets, we employ two techniques in conjunction with DCNN: transfer learning (TL) with fine-tuning and data augmentation tailored specifically for remote sensing imagery. TL allows one to bootstrap a DCNN while preserving the deep visual feature extraction learned over an image corpus from a different image domain. Data augmentation exploits various aspects of remote sensing imagery to dramatically expand small training image data sets and improve DCNN robustness for remote sensing image data. Here, we apply these techniques to the well-known UC Merced data set to achieve the land-cover classification accuracies of 97.8 ± 2.3%, 97.6 ± 2.6%, and 98.5 ± 1.4% with CaffeNet, GoogLeNet, and ResNet, respectively.

Deep convolutional neural networks (DCNNs) have recently emerged as the highest performing approach for a number of image classification applications, including automated land cover classification of high-resolution remote-sensing imagery. In this letter, we investigate a variety of fusion techniques to blend multiple DCNN land cover classifiers into a single aggregate classifier. While feature-level fusion is widely used with deep neural networks, our approach instead focuses on fusion at the classification/information level. Herein, we train three different DCNNs: CaffeNet, GoogLeNet, and ResNet50. The effectiveness of various information fusion methods, including voting, weighted averages, and fuzzy integrals, is then evaluated. In particular, we used DCNN cross-validation results for the input densities of fuzzy integrals followed by evolutionary optimization. This novel approach produces the state-of-the-art classification results up to 99.3% for the UC Merced data set and the 99.2% for the RSD data set.

Object recognition from remote sensing systems is a task of immense interest. With the vast deployment of aerial vehicles and space borne sensors for a wide variety of purposes, it is critical to have robust image processing techniques to analyze massive streams of collected data. Herein, we explore the utility of a feature descriptor learning framework, called improved Evolution-COnstructed (iECO) features. Additionally, an investigation into the combination of iECO features with soft features is conducted. Soft features are a deterministic approach to highlighting pertinent information for improving the quality of features extracted specific to the object of interest while iECO is a way to learn from data the relevant information. Experiments are conducted using four-fold (scene based) cross-validation and are reported in terms of target recognition rates and false alarm rates. Results indicate that iECO features are individually best overall and the combination of iECO and soft features can lead to improved results.

We are publishing an open source library to facilitate the use of three key image processing technologies (GDAL, OpenCV, CUDA) for scalable, high performance geospatial data processing. Herein, we present two computationally demanding algorithms for geospatial data processing which are commonly used for complex structural analysis of imagery. We show that processing time can be reduced by 98.1% and 84.3% for two computationally complex structural analysis algorithms using GPU co-processors over a pure CPU solution. It is our hope that the geoscience community will benefit from, and extend, this library; accelerating the development and integration of novel image processing, pattern recognition, and image information mining techniques.

Numerous fields require large-scale pattern matching to achieve a variety of computational goals. Herein, we present novel graphics processing unit (GPU) extensions that facilitate high-throughput pattern matching in a PostgreSQL database. We have developed an extension framework to perform data block processing of large pattern data sets, using a stream processing design that results in global k-nearest neighbor matches. This framework was specifically designed to support pattern matching on GPU from within the database environment. This approach avoids the necessity of storing an entire data set onto GPU hardware, which facilitates significant scale-up of pattern databases. This provides enormous potential to incorporate or exploit auxiliary (meta)data as part of the pattern matching process, as well as pipelining the results into traditional relational algebra expressions. By pipelining pattern matching results into a relational expression, the power of the database can be leveraged to build result sets based on various parameterized correlations between the query pattern(s) and the results. In this preliminary work, we have integrated GPU-based high-throughput p-norm metric functions into the database server. This allows one to design heterogeneous data processing techniques that combine large-scale content-based image retrieval (CBIR) with traditional data processing capabilities of the database such as relational, spatial, or text search. We present timing characteristics for various pattern sizes and metric combinations, as well as address the balancing of database and GPU parameterization. Our feature vector datasets range from 18 to 85 GB in database table storage size, reaching 100 million 128 dimensional vectors. We are able to efficiently execute global top k searches from within the database.

Geospatial data exists in a variety of formats, including rasters, vector data, and large-scale geospatial databases. There exists an ever-growing number of sensors that are collecting this data, resulting in the explosive growth and scale of high-resolution remote sensing geospatial data collections. A particularly challenging domain of geospatial data processing involves mining information from high resolution remote sensing imagery. The prevalence of high-resolution raster geospatial data collections represents a significant data challenge, as a single remote sensing image is composed of hundreds of millions of pixels. We have developed a robust application framework which exploits graphics processing unit (GPU) clusters to perform high-throughput geospatial data processing. We process geospatial raster data concurrently across tiles of large geospatial data rasters, utilizing GPU co-processors driven by CPU threads to extract refined geospatial information. The framework can produce output rasters or perform image information mining to write data into a geospatial database.

We have developed a fully automated system for change detection of high-resolution satellite imagery. Our system, GeoCDX, is sensor-agnostic, resolution-independent and designed to process the very large volumes of data collected by modern high resolution panchromatic and multispectral imaging satellites. GeoCDX performs fully automated coregistration of imagery; extracts high-level features from the satellite imagery; performs change detection processing to pinpoint locations of change; clusters image tiles to group similar regions of change; and presents results in a variety of ways in an easy-to-use web application that facilitates online discovery, analysis, and dissemination of the change detection results. We applied GeoCDX to 4121 image pairs and successfully coregistered over 91% of the pairs covering a total area greater than 370 000 km2; GeoCDX decreased the average coregistration error from 9.6 ± 8.6 m to 1.8 ± 1.2 m. We show that for some pairs, GeoCDX provides up to a 50% increase in users' efficiency compared to manually performing change detection in common GIS software. Moreover, the change detection assessment performed using GeoCDX was on average four times more accurate compared to the manual approach in large part due to the use of our change intensity map that provides visual cues to the user during exploitation. Finally, change detection analysis using GeoCDX resulted in a missed detection rate of less than 2%.

We present a sliding window-based, per-pixel importance-weighted, multi-scale, cell-structured feature descriptor and demonstrate its performance for recognizing different aircraft from remotely sensed imagery. Opening and closing differential morphological profiles are constructed, then fused with the Choquet integral to create a soft segmentation. A per-pixel importance map is derived from the soft segmentation and used in the calculation of histogram of oriented gradients, local binary patterns, invariant object moments, and Haar-like features. Superiority is demonstrated in comparison to flat single-scale and non-importance weighted representations with encouraging results for both cross-validation and blind testing. Results show that the pyramid, cell-structured, importance-weighting performs better than traditional approaches in the difficult problem space of recognizing objects in remote sensing imagery.

In many large-scale content-based retrieval (CBR) applications, the input to the search process is a complex query that may be composed of several constituent parts. The proposed approach performs CBR queries by breaking down a complex query into several smaller heterogeneous queries. Object-based queries in an imagery search application can be performed by executing a search over several distinct feature space indexes. For example, CBR indexes may exist for spectral, texture, and shape feature vectors extracted from objects. A query for similar objects can be completed by aggregating the results from these multiple indexes. Complementing this concept, a multi-object search can be used to identify relevant groups of objects which match a given set of query objects. For example, a set of objects identified in satellite imagery could be used as a CBR query in order to identify similar groups of objects. Thus, a query can be performed for each object, and these results can be aggregated into multi-object search results by determining the optimal match of the query objects to those in each resulting group. We introduce the absence penalty method and obligatory object query algorithms for performing multi-index and multi-object CBR searches and provide experimental results that show that the proposed approaches efficiently provide search results with a high degree of precision with minimal error. The experimental results shown demonstrate the efficiency and accuracy of the proposed methods; moreover, through the fusion of multi-index and multi-object search techniques, we are able to construct new sophisticated query mechanisms.

In this paper, we present a novel indexing structure that was developed to efficiently and accurately perform content-based shape retrieval of objects from a large-scale satellite imagery database. Our geospatial information retrieval and indexing system, GeoIRIS, contains 45 GB of high-resolution satellite imagery. Objects of multiple scales are automatically extracted from satellite imagery and then encoded into a bitmap shape representation. This shape encoding compresses the total size of the shape descriptors to approximately 0.34% of the imagery database size. We have developed the entropy-balanced bitmap (EBB) tree, which exploits the probabilistic nature of bit values in automatically derived shape classes. The efficiency of the shape representation coupled with the EBB tree allows us to index approximately 1.3 million objects for fast content-based retrieval of objects by shape.

The Geospatial Change Detection and exploitation (GeoCDX) is a fully automated system for detection and exploitation of change between multitemporal high-resolution satellite and airborne images. Overlapping multitemporal images are first organized into 256 m × 256 m tiles in a global grid reference system. The system quantifies the overall amount of change in a given tile with a tile change score as an aggregation of pixel-level changes. The tiles are initially ranked by these change scores for retrieval, review, and exploitation in a Web-based application. However, the ranking does not account for the wide variety of change types that are typically observed in the top-ranked change tiles. To automatically organize the wide variety of change patterns observed in multitemporal high-resolution imagery, we perform tile clustering using the competitive agglomeration (CA) algorithm stabilized using the fuzzy c-means (FCM) algorithm. Each resulting cluster contains tiles with a visually similar type of change. By visual inspection of these tile clusters, GeoCDX users can quickly find certain types of change without having to sift through a large number of tiles initially organized solely by their tile change score, thereby reducing the time it takes for users to discover and exploit the change pattern(s) of greatest interest to a given application (e.g., urban growth, disaster assessment, facility monitoring, etc.). The tile clusters also provide a high-level overview of the various types of change that occur between the two observations. This overview is compared with a similar yet more limited view offered by a relevance feedback tool that requires a user to select sample tiles for use as samples in the reranking process.

The ProgressiveDBSCAN algorithm allows for the progressive clustering of results from a geospatial information retrieval system. Results can be clustered by a combination of both their spatial and non-spatial attributes. The benefit of this clustering is that users are able to sort through the results returned from a geospatial information retrieval system in a spatial context. No longer are results from disparate locations presented to the user, but instead compact spatial clusters are displayed. There is a 98% reduction in the spatial distance between consecutive CBIR results and the spatial distance between the compact clusters; this leads to more efficient analysis of results by reducing the amount of time users spend context switching while on average only adding a few seconds to the query time.

GeoCDX (Geospatial Change Detection and eXploitation) is an integrated system for detecting change between multi-temporal, high-resolution satellite or airborne images. Overlapping images are organized into 256×256 meter tiles in a global grid system. A tile change score measures the amount of change in the tile which is the aggregation of pixel-level change score. The tiles are initially ranked by these change scores. However, this ranking does not account for the wide variety of change types. To learn the change patterns in the data, we apply the fuzzy c-means clustering algorithm to the tiles. Each resulting cluster contains tiles with similar type of change. Users looking for certain types of change can review the tile clusters rather than the more time consuming process of searching through the tile list based on the initial ranking. The clusters also provide users an overview of various types of change found in the scene.

Searching for relevant knowledge across heterogeneous geospatial databases requires an extensive knowledge of the semantic meaning of images, a keen eye for visual patterns, and efficient strategies for collecting and analyzing data with minimal human intervention. In this paper, we present our recently developed content-based multimodal Geospatial Information Retrieval and Indexing System (GeoIRIS) which includes automatic feature extraction, visual content mining from large-scale image databases, and high-dimensional database indexing for fast retrieval. Using these underpinnings, we have developed techniques for complex queries that merge information from heterogeneous geospatial databases, retrievals of objects based on shape and visual characteristics, analysis of multiobject relationships for the retrieval of objects in specific spatial configurations, and semantic models to link low-level image features with high-level visual descriptors. GeoIRIS brings this diverse set of technologies together into a coherent system with an aim of allowing image analysts to more rapidly identify relevant imagery. GeoIRIS is able to answer analysts' questions in seconds, such as "given a query image, show me database satellite images that have similar objects and spatial relationship that are within a certain radius of a landmark."

Today, biomedical media data are being generated at rates unimaginable only years ago. Content-based retrieval of biomedical media from large databases is becoming increasingly important to clinical, research, and educational communities. In this paper, we present the recently developed entropy balanced statistical (EBS) k-d tree and its applications to biomedical media, including a high-resolution computed tomography (HRCT) lung image database and the first real-time protein tertiary structure search engine. Our index utilizes statistical properties inherent in large-scale biomedical media databases for efficient and accurate searches. By applying concepts from pattern recognition and information theory, the EBS k-d tree is built through top-down decision tree induction. Experimentation shows similarity searches against a protein structure database of 53 363 structures consistently execute in less than 8.14 ms for the top 100 most similar structures. Additionally, we have shown improved retrieval precision over adaptive and statistical k-d trees. Retrieval precision of the EBS k-d tree is 81.6% for content-based retrieval of HRCT lung images and 94.9% at 10% recall for protein structure similarity search. The EBS k-d tree has enormous potential for use in biomedical applications embedded with ground-truth knowledge and multidimensional signatures.

Geospatial content-based image retrieval (CBIR) systems can be used to query for visually similar images by identifying similar patterns between a query image and those in the database. When several different classes of features are used, some queries require that each class should be given a different degree of weight; to this end, CBIR indexes are built for each class of features. This paper proposes an approach for weighting multiple indexes in a geospatial CBIR system by mining information from user feedback. After a small number of iterations of relevance feedback and data mining, index weights can be determined dynamically per query. Using this technique geospatial retrieval system precision of results increased from 70% to 79% after 5 iterations of feedback.

This paper presents a framework for the efficient retrieval of satellite imagery from large-scale databases. With the ever-expanding volume of imagery being acquired from satellite platforms it has become increasingly important to locate specific areas of interest within a large database of images. Identifying relevant areas within image databases can be thought of as finding the needle in the haystack problem; too often for a particular task or application there exist a small number of useful images hidden among millions of images. The motivation behind the work presented here is that through the use of geospatial image retrieval systems, the number of scenes that image analysts must manually examine may be decreased dramatically. By using a geospatial image retrieval system as a tool, analysts no longer must manually examine the entire database of imagery, but instead can limit their search to a subset identified by our retrieval system. The techniques that are introduced in this paper have been developed in our image retrieval system named GeoIRIS: Geospatial Information Retrieval and Indexing System.

Discovering relevant knowledge from large-scale geospatial image databases is challenging because of the complexity of describing visual semantics, the computational cost of processing petabytes of data, and the difficulty in summarizing and presenting knowledge. In this paper, we revisit a selective set of core data mining algorithms, namely association rules mining, spatial mining, and temporal mining. We then customize these algorithms using visual content and potential objects extracted from geospatial image databases with other relevant information, such as text-based annotations. Queries utilizing the mining results are also discussed in this paper. These mining and query processing algorithms play an important role in GeoIRIS- Geospatial Information Retrieval and Indexing System.

Indexing protein tertiary structures has been shown to provide a scalable solution for structure-to-structure comparisons in large protein structure retrieval systems. To conduct similarity searches against 53,356 polypeptide chains in a database with real-time responses, two critical issues must be addressed, information extraction and suitable indexing. In this paper, we apply computer vision techniques to extract the predominant information encoded in each 2D distance matrix, generated from 3D coordinates of protein chains. Distance matrices are capable of representing specific protein structural topologies, and similar proteins will generate similar matrices. Once meaningful features are extracted from distance images, an advanced indexing structure, Entropy Balanced Statistical (EBS) k-d tree, can be utilized to index the multidimensional data. With a limited amount of training data from domain experts, namely structural classification of a subset of available protein chains, we apply various techniques in the pattern recognition field to determine clusters of proteins in the multi-dimensional feature space. Our system is able to recall search results in a ranked order from the protein database in seconds, exhibiting a reasonably high degree of precision.

Indexing protein structures has been shown to provide a scalable solution for structure-to-structure comparisons in large protein structure retrieval systems. To conduct similarity searches against 46,075 polypeptide chains in a database with real-time responses, two critical issues must be addressed, information extraction and suitable indexing. In this paper, we apply computer vision techniques to extract the predominant information encoded in each 2D distance matrix, generated from 3D coordinates of protein chains. Distance matrices are capable of representing specific protein structural topologies, and similar proteins will generate similar matrices. Once meaningful features are extracted from distance images, an advanced indexing structure, entropy balanced statistical (EBS) k-d tree, can be utilized to index the multidimensional data. With a limited amount of training data from domain experts, namely structural classification of a subset of available protein chains, we apply various techniques in the pattern recognition field to determine clusters of proteins in the multi-dimensional feature space. Our system is able to recall search results in a ranked order from the protein database in seconds, exhibiting a reasonably high degree of precision.

By utilizing morphological shared-weight neural networks (MSNN) that have been trained for face recognition, common access restriction points can be enhanced to identify particular individuals of interest. A trained MSNN is a computational intelligence structure that learns representation of a specific face that encodes in its connection weights the feature extraction and classification abilities needed to identify an instance of that face. It has been shown effective in analyzing images that contain the target in a group of faces, even with the target face at varying orientations and lighting, as well as occluded target faces. The experiments presented here show the possible application of the MSNN to perform watch-list scanning of faces as individuals pass through access screening areas.