A principled physics-based approach for data augmentation with synthetic aperture sonar (SAS) imagery is proposed. The approach is based on partitioning the wavenumber spectrum of the data. The images that result from retaining only specific sectors of spectral content are referred to as “ghosts.” The approach enables the generation of practically infinite mildly correlated images: high enough that key fundamental features of objects persist, but low enough to engender desired data diversity. The ghosts can be used to help train data-hungry convolutional neural networks (CNNs), but they can also be leveraged at inference time to provide a more robust ensemble prediction that also carries with it a measure of uncertainty. Experimental results on an object classification task with real, measured SAS data highlight the benefits of the approach.

Deep learning has been recently shown to improve performance in the domain of synthetic aperture sonar (SAS) image classification over existing shallow learning solutions. Given the constant resolution with range of a SAS, it is no surprise that deep learning techniques perform so well; the image of the seafloor produced by a SAS system is almost photographic in quality. Despite the image quality benefits of SAS, there is still room for classification improvement particularly in reducing the number of false alarms. This work addresses this by tackling one facet of the classification pipeline: image enhancement. Specifically, we ask and address the following question: Can we train a deep neural network to simultaneously enhance and classify a SAS image? We will respond in the affirmative as we introduce a new deep learning architecture tackling the problem, Data Adaptive Enhancement and Classification Network (DA-ECNet). DA-ECNet is a deep learning architecture which combines image enhancement as part of the classification procedure eliminating the need for a fixed state-of-the-art despeckling algorithm or enhancement module. Additionally, we train both image enhancement and classification jointly resulting in data adaptive image enhancement. Experiments on a challenging real-world dataset reveal that the proposed DA-ECNet outperforms state of the art deep learning as well as traditional feature based methods for SAS image classification.

The value of transferring convolutional neural networks (CNNs) trained with synthetic aperture sonar (SAS) imagery is demonstrated in the context of an underwater unexploded ordnance (UXO) classification task. Specifically, it is shown that CNNs designed for, and trained on, a mine classification task can be transferred across sensors of the same modality - but different frequency bands and sensor resolutions - and also across target concept (from mines to UXO). Importantly, it is shown that this transfer learning outperforms simply training the CNNs "from scratch" using the limited available data that pertains to the ultimate task. A key element underlying this approach is that the CNNs be specially tailored to the particularities of the sensor modality and its data. These findings are valuable because they illustrate how training-data requirements can be eased for data-limited remote-sensing applications.

A general framework for quantifying the worth of a performance-estimation model is proposed. The purpose of the model is to predict the performance of an automatic target recognition algorithm on a given set of test data, while the purpose of the framework is to quantify how well the model fulfills its task. To this end, a quantity referred to as the utility, which is based on the Kullback-Leibler divergence, is introduced. A key aspect of the framework is the inclusion of a significance function that specifies the relative importance of each point in the performance space, here assumed to be defined in terms of false alarm rate and probability of detection. Example significance functions are suggested and discussed. The functionality of the proposed framework is demonstrated on an underwater target detection application involving measured synthetic aperture sonar data. In this context, an image complexity metric is exploited to enable the development of models corresponding to different seafloor conditions and mine-hunting difficulty. The appeal of the framework is its ability to quantitatively assess the utility of competing performance-estimation models and to fairly compare the utility of a model on different test data sets.

A new algorithm called the Mondrian detector has been developed for object detection in high-frequency synthetic aperture sonar (SAS) imagery. If a second (low) frequency-band image is available, the algorithm can seamlessly exploit the additional information via an auxiliary prescreener test. This flexible single-band and multiband functionality fills an important capability gap. The algorithm's overall prescreener component limits the number of potential alarms. The main module of the method then searches for areas that pass a subset of pixel-intensity tests. A new set of reliable classification features has also been developed in the process. The overall framework has been kept uncomplicated intentionally in order to facilitate performance estimation, to avoid requiring dedicated training data, and to permit delayed real-time detection at sea on an autonomous underwater vehicle. The promise of the new algorithm is demonstrated on six substantial data sets of real SAS imagery collected at various geographical sites that collectively exhibit a wide range of diverse seafloor characteristics. The results show that-as with Mondrian's art-simplicity can be powerful.

Deep convolutional neural networks are used to perform underwater target classification in synthetic aperture sonar (SAS) imagery. The deep networks are learned using a massive database of real, measured sonar data collected at sea during different expeditions in various geographical locations. A novel training procedure is developed specially for the data from this new sensor modality in order to augment the amount of training data available for learning and to avoid overfitting. The deep networks learned are employed for several binary classification tasks in which different classes of objects in real sonar data are to be discriminated. The proposed deep approach consistently achieves superior performance to a traditional feature-based classifier that we had relied on previously.

We consider the task of conducting underwater surveys with a sonar-equipped autonomous underwater vehicle (AUV) in environments with strong currents. More specifically, this topic is addressed in the context of mine countermeasure operations employing synthetic aperture sonar (SAS) sensors. Two complementary algorithms that allow the AUV to autonomously adapt its survey route based on sophisticated sensor data it collects in situ, while respecting the unique constraints imposed by the problem, are proposed. The algorithms allow the AUV to (i) adapt its survey heading based on the presence of currents to ensure quality data is collected, and (ii) adapt its survey route to reinspect the most suspicious objects at additional aspects. The flexibility to immediately react in situ to the environmental and tactical conditions sensed during the mission allow the most useful data for object recognition purposes to be collected efficiently. By obviating the recovery and redeployment of the AUV, as well as laboratory-based data-processing during the interregnum, the overall mission timeline can be greatly compressed and operational costs can be reduced. Experimental results illustrating the real-time execution of the proposed algorithms on an AUV are shown for a completely autonomous mission conducted in the North Sea.

The rich content of synthetic aperture sonar (SAS) data is typically used to generate imagery with resolution as high as theoretically possible. But when the ultimate purpose of the imagery is for detecting objects with sizes large compared to the resolution cell, exploiting the raw data in alternative ways can be more useful. We first show how multiple lower-resolution SAS images (i.e., “looks”) can be obtained in an efficient, principled manner by band-limiting the image wavenumber spectrum of a full-resolution SAS image. By combining these multiple looks (from different aspect and frequency bands), a despeckled image that enjoys greater (target) signal to (seabed) reverberation ratio can be produced. On a large set of real SAS data collected at sea, it is demonstrated how better target detection performance of underwater mines can be achieved by using the lower-resolution despeckled imagery rather than single-look, maximum-resolution imagery.

A new unsupervised approach for characterizing seafloor in side-looking sonar imagery is proposed. The approach is based on lacunarity, which measures the pixel-intensity variation, of through-the-sensor data. No training data are required, no assumptions regarding the statistical distributions of the pixels are made, and the universe of (discrete) seafloor types need not be enumerated or known. It is shown how lacunarity can be computed very quickly using integral-image representations, thereby making real-time seafloor assessments on-board an autonomous underwater vehicle feasible. The promise of the approach is demonstrated on high-resolution synthetic-aperture-sonar imagery of diverse seafloor conditions measured at various geographical sites. Specifically, it is shown how lacunarity can effectively distinguish different seafloor conditions and how this fact can be exploited for target-detection performance prediction in mine-countermeasure operations.

This work addresses the task of underwater object recognition in sonar imagery when both human operators and automated algorithms are available. We discuss the issues that have impeded previous attempts at automation, raise key insights related to human perception, present strategies to exploit the skills of humans and computers synergistically, and demonstrate the utility of the proposed approaches on a real object-recognition task employing actual humans acting as operators. Importantly, the strategies outlined here can be immediately adopted in existing (unautomated) target recognition systems with minimal cost, effort, and risk, while still achieving potentially significant performance gains. Moreover, this progress lays the foundation for the acceptance of still-further automated systems in the future. Experimental results are provided from a real mine-search exercise at sea, with recognition performance as a function of human operator effort given for various human-computer divisions of labor.

In many remote-sensing applications, measured data are a strong function of the environment in which they are collected. This paper introduces a new context-dependent classification algorithm to address and exploit this phenomenon. Within the proposed framework, an ensemble of classifiers is constructed, each associated with a particular environment. The key to the method is that the relative importance of each object (i.e., data point) during the learning phase for a given classifier is controlled via a modulating factor based on the similarity of auxiliary environment features. Importantly, the number of classifiers to learn and all other associated model parameters are inferred automatically from the training data. The promise of the proposed method is demonstrated on classification tasks seeking to distinguish underwater targets from clutter in synthetic aperture sonar imagery. The measured data were collected with an autonomous underwater vehicle during several large experiments, conducted at sea between 2008 and 2012, in different geographical locations with diverse environmental conditions. For these data, the environment was quantified by features (extracted from the imagery directly) measuring the anisotropy and the complexity of the seabed. Experimental results suggest that the classification performance of the proposed approach compares favorably to conventional classification algorithms as well as state-of-the-art context-dependent methods. Results also reveal the object features that are salient for performing target classification in different underwater environments.

An algorithm for the in situ adaptation of the survey route of an autonomous underwater vehicle (AUV) equipped with side-looking sonars is proposed. The algorithm immediately exploits the through-the-sensor data that is collected during the mission in order to ensure that quality data is collected everywhere in the area of interest. By introducing flexibility into the survey of the AUV, various limitations of pre-planned surveys are overcome. Experimental results demonstrate the benefit of the proposed approach in terms of higher area coverage in shorter mission times. The signal processing required by the algorithm is fast and computationally efficient such that real-time implementation is feasible. As proof, the proposed adaptive survey approach was implemented on an AUV and executed during a recent live scientific experiment at sea using real, in situ measured data. Results from this experiment are also shown.

A new algorithm for the detection of underwater objects in sonar imagery is proposed. One particularly novel component of the algorithm also detects the presence of, and estimates the orientation of, sand ripples. The overall algorithm is made extremely fast by employing a cascaded architecture and by exploiting integral-image representations. As a result, the method makes real-time detection of objects in streaming sonar data collected by an autonomous underwater vehicle (AUV) feasible. No training data is required because the proposed method is adaptively tailored to the environmental characteristics of the sensed data that is collected in situ. The flexible yet rigorous approach also addresses and overcomes five major limitations that plague the most popular detection algorithms that are in common use. Moreover, the proposed algorithm achieves superior performance across a variety of seabed types on a large, challenging data set of real sonar data collected at sea. Ways to exploit the findings and adapt AUV surveys for optimized detection performance are also suggested.

A new algorithm for performing supervised classification that intentionally alters the training labels supplied with the data set is presented. The proposed approach is motivated by the insight that the average prediction of a group of sufficiently informed people is often more accurate than the prediction of any one supposed expert. This idea that the “wisdom of crowds” can outperform a single expert is implemented in two ways. When labeling error rates can be estimated, sets of labels are drawn as samples from a Bernoulli distribution. When side information is not available, or no labeling errors are suspected, labels are intentionally altered in a structured manner. The framework is demonstrated in the context of an underwater mine classification application on synthetic aperture sonar data collected at sea, with promising results.

This work exploits several machine-learning techniques to address the problem of image-quality prediction of synthetic aperture sonar (SAS) imagery. The objective is to predict the correlation of sonar ping-returns as a function of range from the sonar by using measurements of sonar-platform motion and estimates of environmental characteristics. The environmental characteristics are estimated by effectively performing unsupervised seabed segmentation, which entails extracting wavelet-based features, performing spectral clustering, and learning a variational Bayesian Gaussian mixture model. The motion measurements and environmental features are then used to learn a Gaussian process regression model so that ping correlations can be predicted. To handle issues related to the large size of the data set considered, sparse methods and an out-of-sample extension for spectral clustering are also exploited. The approach is demonstrated on an enormous data set of real SAS images collected in the Baltic Sea.

A model for the detection of sand ripples in synthetic aperture sonar (SAS) imagery is proposed. The approach is based on searching for patterns characterized by three highlight-shadow pairs - corresponding to ripple-wave crests and troughs - at different orientations and at different length-scales. The model also provides an estimate of the orientation of any ripples detected. No training data is required as the underlying physical phenomenon of ripples is modeled directly. The promise of the proposed method is demonstrated on five real, measured SAS images, for which a high probability of (ripple) detection is achieved while maintaining a very low false alarm rate.

A new algorithm for performing classification with imperfectly labeled data is presented. The proposed approach is motivated by the insight that the average prediction of a group of sufficiently informed people is often more accurate than the prediction of any one supposed expert. This idea that the "wisdom of crowds" can outperform a single expert is implemented by drawing sets of labels as samples from a Bernoulli distribution with a specified labeling error rate. Additionally, ideas from multiple imputation are exploited to provide a principled way for determining an appropriate number of label sampling rounds to consider. The approach is demonstrated in the context of an underwater mine classification application on real synthetic aperture sonar data collected at sea, with promising results.

This work addresses the task of designing the optimal survey route that an autonomous underwater vehicle (AUV) should take in mine countermeasures (MCM) operations. It is assumed that the AUV is equipped with a side-looking sonar that is capable of generating high-resolution imagery of the underwater environment. The objective of the path-planning task is framed in terms of maximizing the success of detecting underwater mines in such imagery. Several commonly made - but inaccurate - assumptions about the problem are raised and refuted; it is demonstrated that mine detection performance depends on both range and seabed type. The issue of how to update detection probabilities when multiple views are obtained is also addressed. These various considerations are exploited in conjunction with synthetic aperture sonar (SAS) data to predict detection performance and efficiently design AUV routes that outperform standard ladder surveys. The proposed algorithm can be used to assess and quantify detection performance achieved in past, as well as future, missions. Because the entire route of the AUV can still be designed before deployment, no additional onboard processing or adaptive capabilities are required of the AUV. Therefore, the proposed approach can be immediately applied to systems conducting MCM operations at sea. The method is demonstrated on real SAS imagery collected by an AUV in the Baltic Sea.

An unsupervised seabed segmentation algorithm for synthetic aperture sonar (SAS) imagery is proposed. Each 2 m × 2 m area of seabed is treated as a unique data point. A set of features derived from the coefficients of a wavelet decomposition are extracted for each data point. Spectral clustering is then performed with this data, which assigns the data points to clusters. This clustering result is then used directly to effect a segmentation of the SAS image into different seabed types. Experimental results on four real, measured SAS images demonstrate the promise of the proposed approach. Importantly, accurate image segmentation results are achieved on the large, challenging images without the aid of any training data or parameter estimation.

In this work, we quantify the relationship between synthetic-aperture length (or equivalently, along-track resolution) and seabed segmentation performance experimentally for real synthetic aperture sonar (SAS) imagery. The seabed segmentation algorithm employed uses wavelet-based features, spectral clustering, and a variational Bayesian Gaussian mixture model. It is observed that for this approach, the correct seabed segmentation rate drops approximately ten percentage points for each halving of the along-track resolution between 3 cm and 96 cm. Moreover, changing the along-track resolution has the most significant effect on rocky seabeds.

In many remote-sensing classification problems, the number of targets (e.g., mines) present is very small compared with the number of clutter objects. Traditional classification approaches usually ignore this class imbalance, causing performance to suffer accordingly. In contrast, the recently developed infinitely imbalanced logistic regression (IILR) algorithm explicitly addresses class imbalance in its formulation. We describe this algorithm and give the details necessary to employ it for remote-sensing data sets that are characterized by class imbalance. The method is applied to the problem of mine classification on three real measured data sets. Specifically, classification performance using the IILR algorithm is shown to exceed that of a standard logistic regression approach on two land-mine data sets collected with a ground-penetrating radar and on one underwater-mine data set collected with a sidescan sonar.

A Bayesian data fusion approach for seabed classification using multiview synthetic aperture sonar (SAS) imagery is proposed. The principled approach exploits all available information and results in probabilistic predictions. Each data point, corresponding to a unique 10 m x 10 m area of seabed, is represented by a vector of wavelet-based features. For each seabed type, the distribution of these features is then modeled by a unique Gaussian mixture model. When multiple views of the same data point (i.e., area of seabed) are available, the views are combined via a joint likelihood calculation. The end result of this Bayesian formulation is the posterior probability that a given data point belongs to each seabed type. It is also shown how these posterior probabilities can be exploited in a form of entropy-based active-learning to determine the most useful additional data to acquire. Experimental results of the proposed multiview classification framework are shown on a large data set of real, multiview SAS imagery spanning more than 2 km (2) of seabed.

An image deformation algorithm is integrated with a Gaussian process classifier for application to remote-sensing tasks in which data is in the form of imagery. To combine these disparate techniques, we introduce a novel kernel covariance function for the Gaussian process that allows us to incorporate the result of the image deformation algorithm into a rigorous Bayesian classification framework. The resulting classifier is completely non-parametric in the sense that no parameters or hyperparameters must be learned. The promise of the proposed algorithm is demonstrated on a data set of real, measured land mine data.