Deep neural network (DNN) algorithms have increased dramatically across a wide variety of remote sensing applications. However, as computer vision (CV) researchers have utilized the power of DNN, additional shortcomings have become more clear, including the reliance of DNNs on texture information rather than shape information to perform feature extraction for CV tasks. This lack of shape can negatively impact many remote sensing applications, where shape information can provide the key discriminatory features in object detection. To improve the utilization of shape information in DNNs, a novel model architecture was developed that incorporated shape information extracted using the Differential Morphological Profile (DMP), which was directly embedded in the DNN. While this network showed the ability to outperform traditional DNNs in CV tasks, there existed several instances in which the tradeoff of spectral and shape information proved ineffective. In this study, we present a Hybrid DMP-enabled network that utilizes both the spectral information of traditional DNNs and the shape information extracted by DMP. Our results show that this network is capable of outperforming competing methods in overhead remote sensing object detection.

Earth Observation (EO) data have become abundantly available and at low prices or for free, opening many possibilities to tackle problems such as the mapping of burned areas after wildfires have been extinguished. This task can be completed using deep learning semantic segmentation techniques. In this paper, we first present our methodology for adapting burn area polygons into semantic segmentation training data for deep neural models. Then, we evaluate four deep semantic segmentation models, U-Net, U-Net++, DeepLabV3, and DeepLabV3+ on Sentinel-2 satellite imagery data obtained from the Copernicus program over Canada in 2019. Our results show that DeepLab models outperformed U-Net models with the DeepLabV3 model achieving the best recall of 83.78%, while the DeepLabV3+ achieved the best precision of 84.35%. EO data allows us a faster and more accurate assessment of burned areas, which can accelerate the restoration planning and processing of insurance claims.

Modern satellite sensors can capture reflected optical energy from wavelengths well outside of the range of human perception. Indeed, it is well known that electromagnetic energy from the “non-visible” spectrum can be used to identify materials in geological, oceanographic, and agricultural contexts. What is less understood, however, is if and how this additional spectral information can be leveraged to aid in difficult computer vision tasks, e.g., classification or detection of man-made objects. Thus, we present the results from a series of experiments that evaluate the benefits that Deep Convolutional Neural Networks (DCNN) can garner through the inclusion of more spectral information. For training and testing these networks in image classification, we extract image tiles from the xView multi-spectral dataset. These images contain eight channels of spectral data including five wavelengths bands beyond the three bands traditionally used in computer vision researcher (red, green, and blue). In this work, we report the results of experiments that use an 80–20 train-test split with four DCNN architectures on full 8-band Multispectral Imagery (MSI) and various subsets of this eight banded imagery. The results show that networks trained on MSI have an average testing F1-score around 1.0 point higher than RGB networks trained with the same methods.

Recently, computer vision tasks such as classification and object detection have been dominated by deep neural net-work (DNN) approaches. As DNN methodologies have matured, researchers have found that some of the most common DNN techniques result in models that are highly dependent upon the textures and colors of the imagery, rather than the shape, leading to suboptimal network performance. This problem can be especially problematic in the remote sensing domain, where the discrimination of objects for classification or detection may rely heavily on their shape. To combat this lack of shape bias in DNNs, a network was developed to integrate the Differential Morphological Profile (DMP), an image processing technique for shape extraction, with standard convolutional DNNs for performing computer vision tasks on High Resolution Remote Sensing Imagery (HR-RSI). Previously, this network, known as DMPNet, has been applied to both classification and object detection in HR-RSI with high levels of success. However, the hyper-parametric nature of DMPNet structure required researchers to carefully select the parameters of shape extraction, a choice that could greatly help or hinder DMPNet performance. In this study, we utilize a evolutionary computation algorithm (ECA) to learn the parameters of shape extraction from the data presented to the DMPNet for object detection. Our results show that our DMP-enabled detection models perform better object detection in HR-RSI using an ECA to learn shape extraction parameters than manually selected parameters on the same dataset.

Deep learning has proven to be an immensely powerful tool within the remote sensing space, with capabilities to perform tasks like classification, object detection, and segmentation in a wide range of modalities and spatial resolutions. The deep neural networks (DNN) extract visual features using numerous techniques, including convolutional and pooling layers, residual modules, inception modules, neural architecture search, and attention networks. Researchers have increasingly found that DNN are biased towards texture, and that this bias is a deficiency that when corrected, can boost classification and detection performance of deep learners. A morphology-based network utilizing the differential morphological profile (DMP) as a non-parametric feature extraction layer, known as DMPNet, was proposed with promising results when compared to VGG16. In this work, the original DMPNet is expanded with increased profile depth, as well as alternate convolutional phases, such as ResNet-18 and MobileNet. The resulting architecture shows an ability to increase shape information in the network, and with it, generalizability on high resolution remote sensing imagery.

We are currently experiencing a deluge of high-resolution electroptical (HR-EO) remote sensing images which can be leveraged for a diverse set of applications, ranging from environmental monitoring to defense and security applications. One of the most challenging application domains is the use of machine learning techniques, such as computer vision, for the enhancement and automation of geospatial big data analytics. In this work, we present techniques for extending deep convolutional neural networks (DCNN) from 3-band color imagery to 4-band and 8-band multispectral remote sensing imagery. Performance comparisons are conducted between DCNN for 3-, 4-, and 8-band imagery data using the Functional Map of the World dataset. In particular, we investigate five distinct DCNN architectures for classification, and show that utilizing more input channels of the imagery typically has a positive impact on classification metrics such as F1-score and raw accuracy. Herein, we evaluated two methods for initializing the DCNN convolutional filters with 4-or 8-band inputs via transfer learning from networks trained on 3-band (RGB) images and show these methods are superior to random initialization. Our findings indicate that additional spectral bands have varied benefits, depending on the image class, where challenging classes saw minimal improvements. We detail these findings along with insights into the implications for multispectral DCNN in particular geospatial big data analytics use-cases.

In the last decade, several high resolution remote sensing benchmark datasets have been developed and publicly released. These datasets, while diverse in design, lack the required intra-class variation for high-performing, machine-assisted visual analytics. More specifically, the disparate datasets are suitable for small, closed system evaluation; however they are not well suited for training of computer vision models that are robust in real-world, non-closed environments encountered in true remote sensing applications. To that end, a benchmark meta-dataset (MDS) was developed to facilitate the training of models for machine-assisted visual analytics. Four existing benchmark datasets were combined to build the original MDS, which excelled for training models for both classification and broad area search applications. In this work, we evaluate an enhanced version of the MDS, MDSv2, by integrating co-occurring classes of two additional recently released, publicly available challenge datasets: xView and Functional Map of the World (FMoW). The MDSv2 has 33 classes with 87,470 total samples. We investigate the utility of three neural architecture search (NAS) deep learning architectures on the MDSv2 for both classification and machine-assisted visual analytics. The NAS models trained with the MDSv2 are able to achieve an average F1 of 98.01% and a powerful 0.934 scanning precision.

Deep convolutional neural networks (DCNN) have been the dominant methodology in the field of computer vision over the last decade, using various architectural organizations of successive convolutional layers to extract and assemble low level image features into visual component detectors. One of the tradeoffs that have been made as the community has migrated to deep neural models is the loss of explainability and understanding of which salient visual components are being recognized by a model for a particular task. However, there exists a significant heritage in the remote sensing community that has developed advanced algorithms to analyze the signal and structural characteristics of anthropogenic features. One such approach is the use of morphological image processing techniques to extract objects from imagery and aid in the structural analysis of shapes. In particular, the differential morphological profile (DMP) has had great success extracting object shapes, while naturally grouping the extracted shapes into scale ranges. In this research, we present a novel architecture that integrates an explicit (definable and explainable) scaled object extraction into the network architecture, allowing shallower convolutional layers and lower complexity neural models. The architecture is evaluated on a challenging remote sensing dataset of object classes, providing insights to this approach and illuminating future directions of integrating morphology into neural architectures for enhanced explainability.

Here we demonstrate how Deep Neural Network (DNN) detections of multiple constitutive or component objects that are part of a larger, more complex, and encompassing feature can be spatially fused to improve the detection performance of a larger complex feature. A wide variety of experiments were conducted using the public domain xView dataset to develop and evaluate multiple fusion strategies to improve the detection of Construction Sites using DNN detections of constitutive/component objects commonly associated with construction activity, e.g. cement mixers, dump trucks, etc. The results demonstrate that spatial fusion of multi-scale component object DNN detections can reduce the total detection error rate of Construction Sites by ~30-40%. The best results were obtained when local spatial clustering was used to reduce noise in component vehicle object detections generated by scanning candidate Construction Site locations. This multi-scale spatial fusion approach can be easily extended to improve detection performance in a wide variety of other challenging feature/object search and detection problems in large-scale remote sensing image datasets.

Object detection is a challenging but important task for computer vision, and this is especially true in the remote sensing domain where data collections may bring billions of pixels in a single image. There are many methods for object detection, but in recent years the You Only Look Once (YOLO) algorithm has become a leading technique, gaining popularity due to its ability to perform real-time object detection. While YOLO and its successors have shown excellent results in realtime detection, there are many object detection tasks that require better precision, and do not require real-time detection. In this paper, YOLOv3 is compared to other deep neural networks (DNN) for detecting Vehicle Groups in very high resolution remote sensing imagery (VHR-RSI). A unique centerpoint-based dataset is developed by leveraging a novel data framework and combining quality assured chips with regions of interest in the XView Challenge Dataset. This dataset is then used to train state of the art models including two Neural Architecture Search (NAS) variant DNN for object detection. Additionally, a blind test set is developed to further compare our methods with the YOLOv3 algorithm. The results shows that our method detects vehicle groups with a lower false positive rate (FPR) and better true positive rate (TPR) than state-of-the-art YOLOv3 models for the blind test set; achieving a reduction in error rate of 26.70% over YOLOv3 in F1 Score on the blind test set.

Object detection and localization within high-resolution remote sensing imagery (HR-RSI) is a challenging task for a variety of reasons, such as the complexity and clutter of the image scene and the compactness of the intermixed object classes. Even the most comprehensive training datasets cannot adequately account for the rich diversity and complexity of anthropogenic objects and their contextual settings in large-scale HR-RSI collections. Recent approaches using deep learning techniques include bounding box approaches (e.g., YOLO), object nomination then recognition (e.g., R-CNN), and post-detection object localization of deep neural network detections. Herein, we propose a novel technique that leverages heterogeneous superpixels and deep neural feature extraction to classify the superpixel segmentation through relational analysis. In this preliminary research, we demonstrate the validity of this approach for object detection and localization, as well as its suitability for identifying the irregular shapes of objects (as opposed to a bounding box). Experiments are performed using a sub-set of the xView benchmark dataset with a goal of spearheading future techniques in cluttered scene object recognition that allows deep feature extractors to have more focus on the target objects instead of the surrounding area or nearby object pixels.

Herein, a generalization of the ordered weighted average (OWA) is put forth relative to pattern recognition. The resultant linear order statistic neuron (LOSN) is unique in that it bridges fuzzy sets, specifically fuzzy data/information aggregation, with neural networks. This article discusses the gradient descent-based optimization and geometric interpretation of the LOSN. An advantage is that the LOSN is an efficient shared weight encoding of N! perceptrons, relative to N inputs. Open source codes are provided to facilitate reproducible research. Experiments are conducted to both validate the method and show its non-linear geometric expression.

Recently, many high-resolution remote sensing imagery (HR-RSI) datasets have been released that have diverse characteristics, such as high inter-class and low intra-class variation. Additionally, a benchmark meta-dataset (MDS) was created by agglomerating object classes from multiple HR-RSI datasets. Previous work has shown that deep convolutional neural networks (DCNN) trained on the MDS perform on par with DCNN trained on constituent benchmark datasets in cross-validation experiments. Here we train a model ensemble on four datasets and compare it with a single robust DCNN trained on the MDS. The goal is to better understand under what conditions an ensemble of models, each trained with distinct datasets, is advantaged or disadvantaged compared to a single model trained with the agglomerated MDS for classification performance.

Here we present techniques and algorithms to apply trained deep convolutional neural networks (DCNN) to high-resolution remote sensing imagery datasets covering large areas of the Earth. First, trained DCNN are used to process broad swaths of imagery in an area of interest (AOI) to produce a classification vector response field (CVRF). The CVRF is then aggregated using mode-seeking algorithms to detect potential objects of interest within the AOI. Our research explores the challenges and opportunities of transitioning DCNN out of the training-validation laboratory setting and into the real-world application domain. We show a scalable approach to leverage state-of-the-art DCNN for broad area automated search, detection, and annotation of objects such as tennis courts, storage tanks, runways, and airplanes.

Accurate land cover classification and detection of objects in high-resolution electro-optical remote sensing imagery (RSI) have long been a challenging task. Recently, important new benchmark data sets have been released which are suitable for land cover classification and object detection research. Here, we present state-of-the-art results for four benchmark data sets using a variety of deep convolutional neural networks (DCNN) and multiple network fusion techniques. We achieve 99.70%, 99.66%, 97.74%, and 97.30% classification accuracies on the PatternNet, RSI-CB256, aerial image, and RESISC-45 data sets, respectively, using the Choquet integral with a novel data-driven optimization method presented in this letter. The relative reduction in classification errors achieved by this data driven optimization is 25%-45% compared with the single best DCNN results.