Deep learning (DL)-based fully supervised approaches have demonstrated remarkable performance in sea ice classification, showcasing their potential for highly accurate results. However, their reliance on high-resolution labels poses a formidable challenge, as obtaining such data can be a difficult task. In contrast, our method based on weakly supervised learning excels by operating with lower-resolution polygon labels while still achieving outstanding performance. This approach enables precise pixel-level classification of ice stage of development (SOD) by learning from region-based labels embedded within expert-annotated ice charts. During training, region-based loss functions are introduced to quantify the disparity between predicted tensors describing SOD distributions and label tensors derived from ice charts. We leverage the AI4Arctic Sea Ice Challenge Dataset, comprising over 500 Sentinel-1 synthetic aperture radar (SAR) images, ancillary multisource data, and corresponding ice charts, for model training and evaluation. Visual interpretation and numerical analysis reveal that our weakly supervised method outperforms the fully supervised U-Net benchmark. It yields more accurate SOD predictions, significantly enhancing mapping resolution and class-wise accuracy. This methodology marks a critical step forward in the quest for automated operational sea ice mapping.

Fully supervised deep learning approaches have demonstrated impressive accuracy in sea ice classification, but their dependence on high-resolution labels presents a significant challenge due to the difficulty of obtaining such data. In response, our weakly supervised learning method provides a compelling alternative by utilizing lower-resolution regional labels from expert-annotated ice charts. This approach achieves exceptional pixel-level classification accuracy by introducing regional loss representations during training to measure the disparity between predicted and ice chart-derived sea ice type distributions. Leveraging the AI4Arctic Sea Ice Challenge Dataset, our method outperforms the fully supervised U-Net benchmark in mapping resolution and class-wise accuracy, marking a significant advancement in automated operational sea ice mapping.

Modern object segmentation models are crucial in sports analytics, particularly in dynamic sports like hockey where fast-paced action often results in blurred imagery, such as motion-blurred hockey sticks. Given the shortage of segmentation data for uncommon objects like hockey sticks, data augmentation emerges as a natural solution to enhance training datasets. However, traditional data augmentation methods, which apply transformations at the image level, can distort critical relational cues between objects and their surroundings, undermining a model's ability to accurately segment objects in such challenging conditions. To address this, we propose the Synthetic Local Data Augmentation (SLDA) technique, which selectively applies traditional DA transformations-like scaling, rotation, blurring, and motion blur-directly to individual target objects. This technique allows precise customization of transformations to specifically enhance model robustness against particular types of distortions, such as the motion blur frequently observed with fast-moving hockey sticks. Utilizing a segmented dataset of hockey sticks, SLDA introduces a greater variety of stick instances by inserting elements in the scene with different examples of the same category. This focused approach significantly enhances the model's ability to recognize hockey sticks across a range of visual conditions, thereby improving its generalization capabilities. SLDA detailed experiments in a case study on hockey stick seg-mentation, we demonstrate how SLDA surpasses existing object-level and traditional data augmentation methods in promoting model robustness and adaptive precision. Surpassing alternative by 2.1 %, i.e. from 85% to 87% in F1 Score on small model complexity, and by 5.8%, i.e. from 86% to 92% in mAP50 on large model complexity.

Accurate estimation of human pose and the pose of interacting objects, like a hockey stick, is crucial for action recognition and performance analysis, particularly in sports. Existing methods capture the object along with the human in the bounding boxes, assuming all keypoints are visible within the bounding box. This necessitates larger bounding boxes to capture the object, introducing unnecessary visual features and hindering performance in real-world cluttered environments. We propose a simple image and text-based multimodal solution TokenCLIPose that addresses this limitation. Our approach focuses solely on human keypoints within the bounding box, treating objects as unseen. TokenCLIPose leverages the rich semantic representations endowed by language for inducing keypoint-specific context, even for occluded keypoints. We evaluate the performance of TokenCLIPose on a real-world Ice-Hockey dataset, and demonstrate its generalizability through zero-shot transfer to a smaller Lacrosse dataset. Additionally, we showcase its flexibility on CrowdPose, a popular occlusion benchmark with keypoints within the bounding box. Our method significantly improves over state-of-the-art approaches on all three datasets, with gains of 4.36\%, 2.35\%, and 3.8\%, respectively.

High-resolution sea ice concentration (SIC) maps are critical to support various applications, e.g., climate modeling, ship navigation, and activities in Northern communities. However, operational mapping of SIC based on expert annotations is coarse in spatial resolution and time-consuming to prepare. Although many convolutional neural network (CNN)-based methods have been proposed for automated sea ice mapping from synthetic aperture radar (SAR) imagery in recent years, the lack of pixel-based labels for model training hinders them from producing high-resolution reliable mapping results. To overcome this challenge, this letter presents a novel weakly supervised learning approach that generates pixel-level SIC prediction using coarse region/polygon-level SIC ground truth. Specifically, a novel region-level loss function is designed to enable direct use of regional/polygon SIC values in ice charts for the training of a U-Net-based model. This avoids the errors in transferring region-level SIC values to pixel-level ground-truth SIC values effectively and allows the generation of pixel-level SIC and sea ice extent (SIE) estimates. The proposed approach is evaluated on the recently published AI4Arctic Sea Ice Challenge Dataset with over 500 Sentinel-1 SAR scenes, ancillary data, and associated ice charts. The results demonstrate the effectiveness of the weakly supervised model in producing pixel-level high-resolution SIC maps that are consistent with ice charts and visual interpretation.

This study investigates the relationship between gray-level co-occurrence matrix (GLCM) texture features and synthetic aperture radar (SAR) incidence angle (IA) for sea ice classification. We analyzed dual polarization RADARSAT-2 C-band SAR data comprising 29 scenes. GLCM features were extracted from the radar cross-section (σo) in dB and categorized by sea ice class. To assess IA dependence per sea ice class, we used linear interpolation and the coefficient of determination (R2). We evaluated separability among ice classes using the Jeffries–Matusita distance and confirmed the improved separability with a Bayesian classifier. The results reveal a significant IA dependence of GLCM features. Notably, GLCM features from the HV band display stronger IA dependence and higher separability among ice classes compared to those from the HH band. These findings emphasize the significance of considering IA in the utilization of GLCM features for sea ice classification.

This paper presents an investigation into the use of a heteroscedastic Bayesian neural network (HBNN) for predicting sea ice concentration (SIC) using both passive microwave (PM) and atmospheric data. The primary objective is to provide accurate estimates for downstream services that require uncertainty estimates. To achieve this, HBNNs are implemented using a multilayer perceptron (MLP) architecture with methods for uncertainty quantification based on the Bayes by backprop (BBB) algorithm and a heteroscedastic loss function. The models are trained and tested using data collected from the Eastern Arctic regions. The results of numerical analysis demonstrate that the HBNNs are able to significantly reduce estimation error compared to deterministic NNs. The study also investigates the spatial and seasonal variation of uncertainty in detail.

This paper investigates the influence of input image scale on deep learning-based Beluga whale detection from aerial remote sensing imagery. Beluga whales in the Arctic are jeopardized due to increased coastal activities and climate change. Aerial survey is a common population counting method, and it can be laborious and exhausting to count the number of whales manually. Convolutional neural networks (CNNs) have greatly improved the performance of detecting and counting whales. Since most remote sensing images are very high in resolution, it is a common practice to slice the image into small patches. In this work, we input the full image (after resizing) into an object detection model and compare its performance with the sliding window approach. Experimental results suggest that increasing the input image size helps improve the model’s performance, and the model is able to learn the contextual information.

Although light-weighted explainable deep learning techniques are critical for operational hyperspectral image (HSI) classification, it is very challenging to achieve these techniques due to difficulties to deal with the spatial-spectral complexity and coupling effect in HSI. Leveraging the excellent feature learning capability of the attention mechanism, this paper presents a spatial-spectral dual transformer (SSDT) network that decomposes the conventional spatial-spectral transformer operation into a spatial transformer and a spectral transformer, which not only reduce the model complexity, but also allows the use of self-attention to explain feature relevance. The proposed approach is tested on some benchmark HSI scenes and the results demonstrate that the proposed dual transformer network not only achieves new state-of-the-art performance due to its excellent feature extraction capability, but also enables the analysis and visualization of feature importance and decision making process.

The extraction of building outline vectors is an essential task in supporting various applications. Although the recent development of deep-learning-based techniques has made advancements in the automation of this task, the accuracy and precision are insufficient due to errors caused by abundant noise and obstruction around buildings in aerial images. To better address this issue, this letter presents a new approach called multi-task edge detection (MTED) for building vectorization with the following characteristics. First, instead of detecting building corner points that are very sensitive to noise effects, a deep-learning-based rotated bounding box (RBB) detector is introduced for building edge detection to increase robustness to interference. Second, a multi-task learning strategy is designed to integrate building segmentation inside the METD framework to closely guide edge detection using spatial context. Third, a simple yet effective geometry-guided postprocessing method is designed to reconstruct vectorized building outlines based on the detected edges and learned building shape prior knowledge. The comparative experiments conducted on benchmark very-high-resolution optical aerial images indicate that the proposed approach can significantly outperform the state-of-the-art in terms of vertex-based building outline accuracy metrics. With a test time of 58 ms per building, this method enables efficient building polygon labeling in interactive mapping applications for building surveying and mapping. Code is available athttps://github.com/yifanthomaswu/MTED_framework.

Algorithms designed for ice–water classification of synthetic aperture radar (SAR) sea ice imagery produce only binary (ice and water) output typically using manually labeled samples for assessment. This is limiting because only a small subset of labeled samples are used, which, given the nonstationary nature of the ice and water classes, will likely not reflect the full scene. To address this, we implement a binary ice–water classification in a more informative manner considering the uncertainty associated with each pixel in the scene. To accomplish this, we have implemented a Bayesian convolutional neural network (CNN) with variational inference to produce both aleatoric (data-based) and epistemic (model-based) uncertainty. This valuable information provides feedback as to regions that have pixels more likely to be misclassified and provides improved scene interpretation. Testing was performed on a set of 21 RADARSAT-2 dual-polarization SAR scenes covering a region in the Beaufort Sea captured regularly from April to December. The model is validated by demonstrating: 1) a positive correlation between misclassification rate and model uncertainty and 2) a higher uncertainty during the melt and freeze-up transition periods, which are more challenging to classify. By incorporating the iterative region growing with semantics (IRGS) segmentation algorithm and an uncertainty value-based thresholding algorithm, the Bayesian CNN classification outputs are improved significantly via both numerical analysis and visual inspection.

Unsupervised subpixel mapping (SPM) of hyperspectral image (HSI) is a challenging task due to the difficulties to integrate different prior information and model constraints into a coherent framework. This paper presents a Bayesian neural network for unsupervised HSI SPM, which has the following characteristics. First, the deep image prior (DIP) achieved by a fully convolutional neural network (FCNN) is used to model the spatial correlation efficiently and adaptively in the subpixel label domain. Second, a discrete spectral mixture model (DSMM) is designed to leverage the forward model for enhanced SPM. Third, an auto-encoder architecture is designed to integrate the FCNN and the DSMM to allow efficient unsupervised representational learning using both data and knowledge. Fourth, an expectation-maximization approach is designed to solve the resulting maximum a posteriori problem, where a purified means approach extracts endmembers, and the gradient descent approach updates FCNN parameters for subpixel label estimation. Comparative experiments on both real and simulated HSIs demonstrate that the proposed method outperforms other state-of-the-art methods in terms of both numerical accuracies and visual subpixel mapping results.

Monitoring sea ice in polar regions is essential for environmental modeling and ship navigation. National ice agencies expect robust sea ice classification methods for operational use. However, fully supervised machine learning models require large training datasets, which are usually limited to the sea ice classification domain. Therefore, a semi-supervised sea ice classification model is proposed to address this challenge. First, the IRGS segmentation is applied to generate superpixels that construct the graph. Then, two graph convolutional layers are utilized to learn the features of each node. Finally, a softmax layer assigns labels to the nodes in the graph. The proposed model is named IRGS-GCN and tested on four RADARSAR-2 dual-polarized scenes. The experimental results show that the IRGS-GCN achieves an overall accuracy of 95.17% and outperforms fully-supervised random foreset and ResNet trained on limited data. Most of the sea ice boundary and leads are successfully preserved in the results.

Automatic building footprint extraction from remote sensing imagery is a challenging task with important applications in geomatics and environmental science. Significant advances have been made in this field as a result of the emergence of deep convolutional neural networks (CNNs) designed for semantic segmentation. Although CNNs have demonstrated state-of-the-art performance in coarse annotation and identification of buildings, the accuracy of extracted building footprints is still insufficient for high-precision applications such as mapping and navigation. We propose the topography-aware multi-resolution fusion learning strategy tailored to the problem of enhanced building footprint extraction. More specifically, we introduce a topography-aware loss (TAL) for enhancing a deep CNN’s ability to learn heterogeneous building features for better boundary preservation during segmentation. We then incorporate the proposed TAL loss within a multi-resolution fusion architecture to boost high-resolution segmentation performance. Finally, we introduce a novel metric named average thresholded contour accuracy (tCA) which specifically measures the accuracy of segmentation boundaries. The experimental results on the SpaceNet buildings dataset show significant improvements in boundary integrity of extracted building footprints when compared with previously proposed methods. Hence, this method enables accurate boundary annotation toward automatic production of building footprint maps for high-precision applications.

Spectral unmixing (SU) plays a fundamental role in hyperspectral image (HSI) processing. Effective SU relies on the accurate and efficient characterization of the noise effect, the endmembers, and the spatial correlation effect in abundances, as well as efficient optimization techniques to estimate these effects. To address these issues, this article presents a Bayesian fully convolutional hyperspectral unmixing network (BCUN) with the following key characteristics. First, a fully convolutional neural network (FCNN)-based deep image prior (DIP) is designed for enhanced characterization and estimation of the spatial context information in abundance maps, leading to more efficient and accurate abundance modeling than the traditional nonnegative least squares (NNLS) approaches. Second, a multivariate Gaussian distribution with an anisotropic covariance matrix is designed to characterize the conditional distribution of the spectral observations, leading to a novel Mahalanobis distance-based loss for FCNN training that is better capable of addressing the noise heterogeneous effect in HSI than the Euclidean distance-based mean squared error (MSE) loss in traditional deep neural networks. Third, the designed conditional distribution of spectral observations also enables the incorporation of the spectral mixture model (SMM) into the FCNN training process for effectively leveraging the knowledge in the forward spectral model. Fourth, the endmembers are modeled and estimated by a “purified means” approach that is capable of better characterizing endmembers. Finally, the above key components are coherently integrated into a Bayesian framework, and the resulting maximuma posteriori(MAP) problem is solved by a designed expectation–maximization (EM) algorithm. Experimental results on both simulated and real HSIs demonstrate that the proposed BCUN approach outperforms the other classical and state-of-the-art methods on both endmember estimation and abundance estimation.

Sentinel-1 is a synthetic aperture radar platform that provides free and open-source images of the Earth. A product type of Sentinel-1 is ground range detected (GRD), which records intensity while discarding phase information from the radar backscatter. Especially in cross-polarized GRD images, there are noticeable intensity changes throughout the image that are caused by amplifying the noise floor of the signal, which varies due to the nonuniform radiation pattern of the satellite’s antenna. While Sentinel-1 has instrument processing facility (IPF) software to estimate the noise floor, even in the newer versions (3.1 or above) of the IPF software there are still instances where the estimates provided do not fit the actual noise floor in the image, which is particularly noticeable in transitions between adjacent subswaths. In this work, we propose a method that reduces the impact of the varying noise-floor throughout the image. The method models the intensity of the noise floor to be a power function of the radiation pattern power. The method divides the swath into several sections depending on the location of the local minimum and maximum of the radiation pattern power with respect to the range. The parameter estimation is portrayed as a geometric programming problem that is transformed into a linear programming problem by logarithmic transformation. Affine offsets are computed for each subswath by a weighted least squares approach. Vast improvement is found on extra-wide (EW) and interferometric wide (IW) Sentinel-1 modes over cross-polarized images. Code implementation is available athttps://github.com/PeterQLee/sentinel1_denoise_rs.

Although accurate training and initialization information is difficult to acquire, unsupervised hyperspectral subpixel mapping (SPM) without relying on this predefined information is an insufficiently addressed research issue. This letter presents a novel Bayesian approach for unsupervised SPM of hyperspectral imagery (HSI) based on the Markov random field (MRF) and a band-weighted discrete spectral mixture model (BDSMM), with the following key characteristics. First, this is an unsupervised approach that allows adjustment of abundance and endmember information adaptively for less relying on algorithm initialization. Second, this approach consists of the BDSMM for accommodating the noise heterogeneity and the hidden label field of subpixels in HSI. The BDSMM also integrates SPM into the spectral mixture analysis and allows enhanced SPM by fully exploring the endmember-abundance patterns in HSI. Third, the MRF and BDSMM are integrated into a Bayesian framework to use both the spatial and spectral information efficiently, and an expectation-maximization (EM) approach is designed to solve the model by iteratively estimating the endmembers and the label field. Experiments on both simulated and real HSI demonstrate that the proposed algorithm can yield better performance than traditional methods.

Synthetic aperture radar (SAR) sea ice imagery is a promising source of data for sea ice data assimilation. Classification of SAR sea ice imagery into ice and water is of particular relevance due to its relationship with ice concentration, a key variable in sea ice data assimilation systems. With increasing volumes of SAR data, automated methods to carry out these classifications are of particular importance. Although several automated approaches have been proposed, none look at the impact of including an estimate of uncertainty of the model parameters and input features on the classification output. This article uses an established database of SAR image features to train a multilayer perceptron (MLP) neural network to classify pixel locations as either ice, water, or unknown. The classification accuracies are benchmarked using a recently developed logistic regression approach for the same database. The two methods are found to be comparable. The MLP approach is then enhanced to allow uncertainty to be estimated at each pixel location. Following methods proposed in the deep learning community, two kinds of uncertainty are considered. The first, epistemic uncertainty, is that due to uncertainty in the MLP weights. The second kind of uncertainty, aleatoric uncertainty, is that which cannot be explained by the model, and is therefore associated with the input data. It is found that including these uncertainties in the MLP models reduces their accuracies slightly, but also reduces misclassification rates. This is of particular importance for data assimilation applications, where misclassifications could severely degrade the analysis.

The Canadian RADARSAT Constellation Mission (RCM) proposes a new synthetic aperture radar (SAR) data mode called compact (hybrid or partial) polarimetry (CP) in a wide swath. Compact polarimetry maximizes the measurement potential if the multilook complex (MLC) coherence matrix of the SAR backscattered field is used. The MLC CP coherence matrix follows the Wishart distribution. In this paper, an unsupervised region-based semantic segmentation of the MLC CP coherence matrix data using the complex Wishart distribution is presented. The segmentation method is an extension of the iterative region growing with semantics (IRGS) to complex CP data. The proposed algorithm is called CP-IRGS and is formulated based on conditional random fields (CRFs) incorporating edge strength over the image. Applications of the algorithm are demonstrated using a simulated MLC CP data set and a real single-look complex (SLC) quadrature polarimetric (QP) SAR data set which is used to derive the MLC CP data.

Synthetic aperture radar images from the Sentinel-1 program are obtained by interpreting signals from a non-uniform radiation pattern. Cross-polarized images in extra-wide mode show significant additive noise patterns that take the form of varying intensity and are independent of the ground targets. While Sentinel-1 provides a method for removing these noise patterns via noise-calibration files, there still remain significant issues in the transformed products, particularly from discontinuous changes among adjacent subswaths. In this work, we consider recalibration by assuming the noise-gain to be a power function of the platform's radiation pattern. We propose a method that estimates the scaling and exponent parameters of the power-function by applying linear programming to the log transform of the data and the radiation pattern intensity, alone with affine rescaling with least-squares estimation. Our method is able to rescale Sentinel-1 scenes to have a more consistent intensity profile among the subswaths of the image.

In this paper, we address the hyperspectral image (HSI) classification task with a generative adversarial network and conditional random field (GAN-CRF)-based framework, which integrates a semisupervised deep learning and a probabilistic graphical model, and make three contributions. First, we design four types of convolutional and transposed convolutional layers that consider the characteristics of HSIs to help with extracting discriminative features from limited numbers of labeled HSI samples. Second, we construct semisupervised generative adversarial networks (GANs) to alleviate the shortage of training samples by adding labels to them and implicitly reconstructing real HSI data distribution through adversarial training. Third, we build dense conditional random fields (CRFs) on top of the random variables that are initialized to the softmax predictions of the trained GANs and are conditioned on HSIs to refine classification maps. This semisupervised framework leverages the merits of discriminative and generative models through a game-theoretical approach. Moreover, even though we used very small numbers of labeled training HSI samples from the two most challenging and extensively studied datasets, the experimental results demonstrated that spectral-spatial GAN-CRF (SS-GAN-CRF) models achieved top-ranking accuracy for semisupervised HSI classification.

Although efficient hyperspectral image (HSI) denoising relies on complete and accurate description and modeling the spatial-spectral signal in HSI, the current approaches do not fully account for key characteristics of HSI, i.e., the mixed spectra effect, the spatial nonstationarity effect, and noise variance heterogeneity effect. To address this issue, this article presents a linear spectral mixture model with nonlocal means constraint (LSMM-NLMC), with the following advantages. First, LSMM-NLMC can effectively learn the signal in mixed pixels in HSI by estimating clean endmembers and abundances for image restoration. Second, LSMM-NLMC can efficiently address nonstationary spatial correlation effect by imposing NLMC on the latent scene signal. Last, LSMM-NLMC provides accurate noise characterization by accounting for noise variance heterogeneity effect using a band-dependent noise model and a band-weighted Mahalanobis distance for similarity measurement. A novel optimization method based on the expectation-maximization (EM) algorithm and the purified means approach is used to efficiently solve the resulting maximum a posterior (MAP) problem. The experiments on both simulated and real HSI data sets demonstrate that the visual quality and denoising accuracy are significantly improved by the proposed LSMM-NLMC compared with previous methods.

Despeckling of synthetic aperture radar (SAR) is a known research challenge. A novel solution to this problem has been developed and evaluated via an iterative maximum a posterior estimation incorporating a Bayesian joint decorrelation and despeckling based on a correlation model. This model realistically explores the physical correlation process of SAR speckle noise and is determined automatically via Bayesian estimation in the log-Fourier domain. A patchwise computation is used to account for the spatial nonstationarity associated with SAR image data. The proposed approach is compared to the existing despeckling techniques using both simulated and real SAR data, and the experimental results demonstrate the improvement in preserving the structural details while suppressing speckle noise.

Automatic classification methods using satellite imagery are beneficial in the sea-ice-type mapping of the Arctic regions. In the near future, the RADARSAT Constellation Mission (RCM) will be launched, providing unique compact polarimetric (CP) synthetic aperture radar (SAR) data, expected to be an improvement over the current RADARSAT-2 dual-polarimetric SAR imagery. This motivates the implementation of a CP-dedicated automatic scene classification approach. First, an existing unsupervised segmentation algorithm called iterative region growing using semantics (IRGS) is used to segment ice-class homogeneous regions to reduce the impact of speckle noise. Second, a support vector machine (SVM) is used to classify the ice-type labels for each homogeneous region. Two complex quad-polarimetric RADARSAT-2 scenes are used to mathematically simulate the corresponding CP scenes for algorithm testing. Classification accuracy shows that using only the two CP intensity images leads to improved results compared with standard dual-polarimetric scenes. Using the CP data, the best classification results are obtained with the reconstructed QP data for the IRGS segmentation and all derived CP features for the SVM labeling. The results support the expected potential that CP scenes will provide improved sea-ice classification than the current operational dual-pol scenes.

The problem of limited labeled training samples is challenging for the classification of remote sensing imagery. We develop a joint classification and segmentation algorithm to address this problem. Our algorithm combines semisupervised learning and conditional random fields (CRFs) into a single framework. The multimodal Gaussian maximum-likelihood classifier is used to estimate the probabilities for the unary potentials of the CRF. Unlike traditional methods based on random fields, region merging is concatenated with the CRF inference to reduce the number of nodes iteratively. Moreover, a semisupervised technique called self-training is used, which iteratively enlarges the training sample set and retrains the classifier. The selection of training samples is based on the region information, so that the risk of assigning wrong labels is largely reduced. The proposed algorithm is applied to hyperspectral image classification, and results on benchmark data sets show that the proposed algorithm significantly improves classification performance after using self-training, and outperforms state-of-the-art spectral-spatial methods for limited labeled training samples.

We propose a fully convolutional neural network (FCNN) model for ice concentration estimation from dual-polarized SAR images. Our network contains 5 convolutional layers. Tested in the Gulf of Saint Lawrence during freeze-up, the proposed model is demonstrated to generate improved ice concentration estimates compared to a CNNs with similar structure.

Although the synthesis of the synthetic aperture radar (SAR) imagery with both sea ice and oil spills can significantly benefit in improving the consistency and comprehensiveness of testing and evaluating algorithms that are designed for mapping cold ocean regions, creating such imagery is difficult due to the heterogeneity and complexity of the source images. This letter presents an enhanced region-based probabilistic posterior sampling approach to effectively synthesize SAR imagery with different ocean features. In the proposed approach, instead of relying entirely on the SAR intensity values, the posterior sampling is performed based on a number of quantitative factors, such as intensity, label field, and the prior class probability of sampling candidates, constituting a complete probabilistic framework that addresses key aspects in the synthesis of SAR imagery from heterogeneous sources. The experiments demonstrate that the proposed approach can better address the difficulties caused by the heterogeneity in the source images compared with the existing state-of-the-art ice synthesis method, and it will improve the consistency, comprehensiveness, and fairness of the evaluation of the remote sensing classification and segmentation algorithms.

The classification of pixels in remotely sensed imagery (RSI) into land cover classes typically requires knowing the labels of some image pixels for model training. However, accurate pixel-level label information is usually difficult and expensive to acquire, which restricts the applicability of supervised image classification methods. In contrast, the region labels information that specifies which classes are contained in a region of the image that is easier to acquire and less susceptible to identification errors. To utilize the region label information for remotely sensed image classification, this paper presents a weakly supervised image classification approach using label constraint and edge penalty (ILCEP), which has the following key characteristics. First, the predefined region labels are used as constraints in ILCEP to guide the inference of pixel labels in the image. Second, the edges between neighboring pixels are used as penalties to address the spatial contextual information in the image. Third, the label constraint and edge penalty are incorporated into the conditional random field framework, and simultaneous model learning and label inference are achieved by solving the maximum a posteriori problem through an enhanced simulated annealing algorithm. Experiments on both simulated and real RSIs demonstrate that the proposed approach can achieve high classification accuracy by knowing only the region-level label information.

With the fast advancement of remote sensing platforms and sensors, remotely sensed imagery (RSI) is increasingly being characterized by both high spatial resolution and high temporal resolution. How to efficiently use the rich spatial and temporal information in RSI for highly accurate object detection and classification is an important research question. Nevertheless, there is still a lack of a probabilistic framework that is capable of fully accounting for the spatial-temporal information in RSI for improved applications. In this paper, we present a Bayesian spatial-temporal random field model that constitutes a complete probabilistic framework for fully explaining the spatial-temporal correlation in RSI, leading to an enhanced object detection approach that is used for cloud detection from RSI. Under the Bayesian theorem, the posterior distribution of a label field is decomposed into the label prior, the data likelihood, the temporal label likelihood, and the temporal data likelihood. To address the difficulties in modeling the complex spatial-temporal correlation effect in the temporal data likelihood, a stochastic sampling approach is presented. Based on the maximum a posteriori approach, the posterior distribution is seamlessly integrated into the graph-cut optimization framework, and, therefore, the model optimization can be efficiently solved. The proposed algorithm is tested for cloud detection on both simulated and real RSIs and the results demonstrate that the proposed algorithm can effectively exploit the spatial-temporal information for achieving higher detection accuracy.

Projecting an image onto an arbitrary non-flat screen surface leads to undesired geometric distortions in the image's projection. Geometric correction pre-distorts the image being projected such that the image's projection appears geometrically correct. In this work, we propose a novel saliency-guided projection geometric correction (SPGC) method that leverages calibration parameters along with 3D surface geometry captured by the projector-camera system to compensate for the geometric distortions created by non-flat screen surfaces. The proposed SPGC method incorporates a novel sampling scheme that selects a small set of surface points for geometric correction estimation based on local surface saliency, which greatly reduces the computational complexity of geometric correction estimation process. Experimental results using a test non-flat screen surface with abrupt edges and curve showed that the proposed SPGC approach achieved superior distortion compensation performance both quantitatively and qualitatively when compared to an unguided projection geometric correction method, while requiring just 3% of the samples used by a conventional densely-sampled projection geometric correction method.

A novel method, Stochastically Acquired Photoplethysmo-gram for Heart rate Inference in Realistic Environments (SAPPHIRE), is proposed for robust remote heart rate measurement through broadband video. A set of stochastically sampled points from the cheek region is tracked and used to construct corresponding time series observations via skin erythema transforms. From these observations, a photo-plethysmogram (PPG) waveform is estimated via Bayesian minimization, with the required posterior probability inferred using a Monte Carlo approach. To mitigate the effects of noise, the contribution of each observation is weighted based on the observation's likelihood to contain relevant data. A bandpass filter is applied to the estimated PPG waveform to omit implausible heart rate frequencies, and the heart rate is estimated through frequency domain analysis. Experimental results acquired from a set of thirty videos indicate significantly improved performance in comparison to state-of-the-art methods.

A novel approach to spatio-temporal saliency detection in video is proposed. Saliency computation is considered as an optimization problem that maximizes the energy of a fully-connected graphical model based on spatio-temporal feature distinctiveness. Each pixel in a video is modeled by a node, and the spatio-temporal feature distinctiveness between pixels by edges connecting the nodes in the graph. The computational complexity is addressed by compressing the fully-connected graph into an abstracted, fully-connected graph with far fewer nodes, where each node in the new graph characterizes nodal groups. The saliency value of each pixel is then computed based on spatio-temporal feature distinctiveness and the energy representation of its nodal group given the constructed graphical model. Experimental results show that our approach outperforms existing approaches to spatio-temporal salient region detection.

In medical image analysis, multi-modal registration has been a challenging task due to the complex intensity relationship between images to be aligned. Conventional multi-modal approaches tend to assess the accuracy of the alignment by measuring a similarity based on statistical dependency of the intensity values between images. However, measuring statistical similarity measures, such as mutual information, is not promising, especially in those cases with complex and spatially dependent intensity relations. A new similarity measure is proposed based on the concept of self-similarity, the similarity of patches within an image, motivated by the fact that similar structures are more probable to undergo similar intensity transformations. The method is applied to the registration framework to align simulated and real brain images from different modalities and compared to the conventional multi-modal registration method. Quantitative evaluation of the method demonstrates that better accuracy can be achieved.

The potential of active learning (AL) methods for improving the marine oil spills identification system is exploited using 10-year (2004-2013) RADARSAT data. Six basic AL methods are proposed according to the uncertainty criteria and coupled with the support vector machine (SVM) classifier. As many as 56 commonly used features are used for the classification. The AUC measures are estimated using the 6-fold cross validation technique to achieve bias-reduced evaluation of performance of the AL-based classifiers. The experiment results show that 22 to 74 percent of samples could be reduced for training SVM classifiers with certain destination performance, if the proper AL method such as AL-6 is selected and the criteria of exploitation and exploration could further improve the performance.

High-resolution ice concentration maps are of great interest for ship navigation and ice hazard forecasting. In this case study, a convolutional neural network (CNN) has been used to estimate ice concentration using synthetic aperture radar (SAR) scenes captured during the melt season. These dual-pol RADARSAT-2 satellite images are used as input, and the ice concentration is the direct output from the CNN. With no feature extraction or segmentation postprocessing, the absolute mean errors of the generated ice concentration maps are less than 10% on average when compared with manual interpretation of the ice state by ice experts. The CNN is demonstrated to produce ice concentration maps with more detail than produced operationally. Reasonable ice concentration estimations are made in melt regions and in regions of low ice concentration.

Unsupervised feature extraction from hyperspectral images (HSIs) relies on efficient data representation. However, classical data representation techniques, e.g., principal component analysis and independent component analysis, do not reflect the intrinsic characteristics of HSI, and as such, they are less efficient for producing discriminative features. To address this issue, we have developed an intrinsic representation (IR) approach to support HSI classification. Based on the linear spectral mixture model, the IR approach explains the underlying physical factors that are responsible for generating HSI. Moreover, it addresses other important characteristics of HSI, i.e., the noise variance heterogeneity effect in the spectral domain and the spatial correlation effect in image domain. The IR model is solved iteratively by alternating the estimation of IR coefficients given IR bases and the update of IR bases given the coefficients. The resulting IR coefficients are discriminative, compact, and noise resistant, thereby constituting powerful features for improved HSI classification. The experiments on both simulated and real HSI demonstrate that the features extracted by the IR model are more capable of boosting the classification performance than the other referenced techniques.

Local saliency models are a cornerstone in image processing and computer vision, used in a wide variety of applications ranging from keypoint detection and feature extraction, to image matching and image representation. However, current models exhibit difficulties in achieving consistent results under varying, non-ideal illumination conditions. In this paper, a novel texture-illumination guided energy response (TIGER) model for illumination robust local saliency is proposed. In the TIGER model, local saliency is quantified by a modified Hessian energy response guided by a weighted aggregate of texture and illumination aspects from the image. A stochastic Bayesian disassociation approach via Monte Carlo sampling is employed to decompose the image into its texture and illumination aspects for the saliency computation. Experimental results demonstrate that higher correlation between local saliency maps constructed from the same scene under different illumination conditions can be achieved using the TIGER model when compared to common local saliency approach, i.e., Laplacian of Gaussian, Difference of Gaussians, and Hessian saliency models.

This paper proposes DESIRe (Discontinuous Energy Seam-carving Image Retargeting), an improved seam carving approach for content-aware image retargeting. The proposed algorithm introduces a novel discontinuous seam carving optimization process that incorporates not only a structural energy cost functional, but also a texture energy cost functional to handle scenes with complex structural and textural characteristics. Experimental results using a variety of scenes with dense image detail characteristics show that the proposed DESIRe seam carving approach can provide improved visual quality when compared to a number of existing seam carving methods. These results illustrate the potential of the proposed DESIRe approach for improving content-aware image retargeting that reduces the loss of important image information without causing significant visual distortions or artifacts.

This paper proposes DESIRe (Discontinuous Energy Seam-carving Image Retargeting), an improved seam carving approach for content-aware image retargeting. The proposed algorithm introduces a novel discontinuous seam carving optimization process that incorporates not only a structural energy cost functional, but also a texture energy cost functional to handle scenes with complex structural and textural characteristics. Experimental results using a variety of scenes with dense image detail characteristics show that the proposed DESIRe seam carving approach can provide improved visual quality when compared to a number of existing seam carving methods. These results illustrate the potential of the proposed DESIRe approach for improving content-aware image retargeting that reduces the loss of important image information without causing significant visual distortions or artifacts.

Despeckling of complex polarimetric synthetic aperture radar (SAR) images is more difficult than denoising of general images due to the low signal-to-noise ratio and the complex signals. A novel stochastic polarimetric SAR despeckling technique based on quasi Monte Carlo sampling (QMCS) and region-based probabilistic similarity likelihood has been developed. The despeckling of complex polarimetric SAR images is formulated as a Bayesian least squares optimization problem, where the posterior distribution is estimated by QMCS in a nonparametric manner. The QMCS approach allows the incorporation of the statistical description of local texture pattern similarity. Experiments on two benchmark quad-pol SAR images demonstrate that the proposed QMC texture likelihood sampling (QMCTLS) filter outperforms referenced methods in terms of both noise removal and detail preservation.

A feature extraction approach for hyperspectral image classification has been developed. Multiple linear manifolds are learned to characterize the original data based on their locations in the feature space, and an ensemble of classifier is then trained using all these manifolds. Such manifolds are localized in the feature space (which we will refer to as “localized manifolds”) and can overcome the difficulty of learning a single global manifold due to the complexity and nonlinearity of hyperspectral data. Two state-of-the-art feature extraction methods are used to implement localized manifolds. Experimental results show that classification accuracy is improved using both localized manifold learning methods on standard hyperspectral data sets.

We propose a simple yet effective structure-guided statistical textural distinctiveness approach to salient region detection. Our method uses a multilayer approach to analyze the structural and textural characteristics of natural images as important features for salient region detection from a scale point of view. To represent the structural characteristics, we abstract the image using structured image elements and extract rotational-invariant neighborhood-based textural representations to characterize each element by an individual texture pattern. We then learn a set of representative texture atoms for sparse texture modeling and construct a statistical textural distinctiveness matrix to determine the distinctiveness between all representative texture atom pairs in each layer. Finally, we determine saliency maps for each layer based on the occurrence probability of the texture atoms and their respective statistical textural distinctiveness and fuse them to compute a final saliency map. Experimental results using four public data sets and a variety of performance evaluation metrics show that our approach provides promising results when compared with existing salient region detection approaches.

Multi-atlas label fusion is a widely used approach in medical image analysis that has improved the accuracy of segmentation. Majority voting, as the most common combination strategy, weighs each candidate in the atlas database equally. More sophisticated methods rely on the intensity similarity of each atlas to the target volume. However, these methods cannot handle those cases in which the atlases and the target image are in different modalities. A new method for label fusion is proposed, based on a structural similarity measure, relying on the structural relationships of features extracted from an undecimated wavelet transform instead of explicit image intensities. The new label fusion method has been tested on simulated and real MR images; segmentation results are promising, and open the door to a wider range of multi-modal approaches.

In this paper, unsupervised segmentation methods are investigated for surficial materials mapping in Nunavut, Canada. Different satellite data sources including RADARSAT-2 polarimetric image, LANDSAT-7 image, and DEM data are combined and three unsupervised segmentation methods are compared. Results show that IRGS has better performance than the other two methods.

Feature space projection, or feature projection is an active research topic in machine learning. Some projection methods have been used in remote sensing for dimension reduction, especially for hyperspectral data due to high dimensionality. Projection methods can improve the performance of classifiers susceptible to the Hughes phenomenon. However, the effect of feature projection for more advanced classifiers has not been well-studied, and there are few studies comparing projection methods for hyperspectral image classification. A comprehensive study has been performed on the effect of feature projection for classification using both reduced and full dimensions. The performance of six feature projection methods (PCA, LLE, LDA, LFDA, LMNN, and SPCA) using three classifiers has been explored on three hyperspectral data sets. Results show that the performance of feature projection methods on different classifiers are mainly consistent for different data sets. LFDA achieves the best overall performance considering all data sets and all classifiers.

Classification of hyperspectral imagery using too few labeled samples is a challenging problem considering the high dimensionality of hyperspectral imagery. In this paper, an ensemble method combining rotation forests and AdaBoost is proposed to tackle this problem. By adaptive boosting, AdaBoost can significantly reduce classification error in an iteration compared to a single classifier, and the rotation matrix can increase diversity so that the ensemble performance can be further improved. Experimental resutls show that the final classification accuracy of the proposed algorithm consistently outperforms other state-of-the-art classification methods.

A two-layer feed forward neural network is used to estimate ice concentration from SAR images directly in this research. SAR image patches are used as input. The CIS (Canadian Ice Service) ice concentration image analyses are used to train the neural network. The experiment shows that the simple neural network can be used to generate a reasonable ice concentration with no preprocessing to the SAR images.

Speckle noise is found in synthetic aperture radar (SAR) images and can affect visualization and analysis. A novel stochastic texture-based algorithm is proposed to suppress speckle noise while preserving the underlying structural and texture detail. Based on a sorted local texture model and a Fisher-Tippett logarithmic-space speckle distribution model, a Monte Carlo texture likelihood sampling strategy is proposed to estimate the true signal. The algorithm is compared to six other classic and state-of-the-art despeckling techniques. The comparison is performed both on synthetic noisy images added and on actual SAR images. Using peak signal-to-noise ratio, contrast-to-noise ratio, and structural similarity index as image quality metrics, the proposed algorithm shows strong despeckling performance when compared to existing despeckling algorithms.

Mapping ice and open water in ocean bodies is important for numerous purposes, including environmental analysis and ship navigation. The Canadian Ice Service (CIS) has stipulated a need for an automated ice-water discrimination algorithm using dual polarization images produced by RADARSAT-2. Automated methods can provide mappings in larger volumes, with more consistency, and in finer resolutions, which are otherwise impractical to generate. We have developed such an automated ice-water discrimination system called MAp-Guided Ice Classification. First, the HV (horizontal transmit polarization, vertical receive polarization) scene is classified using the “glocal” method, i.e., a hierarchical region-based classification method based on the published iterative region growing using semantics (IRGS) algorithm. Second, a pixel-based support vector machine (SVM) using a nonlinear radial basis function kernel classification is performed exploiting synthetic aperture radar gray-level cooccurrence texture and backscatter features. Finally, the IRGS and SVM classification results are combined using the IRGS approach but with a modified energy function to accommodate the SVM pixel-based information. The combined classifier was tested on 20 ground truthed dual polarization RADARSAT-2 scenes of the Beaufort Sea containing a variety of ice types and water patterns across melt, summer, and freeze-up periods. The average leave-one-out classification accuracy with respect to these ground truths is 96.42%, with a minimum of 89.95% for one scene. The MAGIC system is now under consideration by the CIS for operational use.

Active contours are a popular approach for object segmentation that uses an energy minimizing spline to extract an object's boundary. Nonparametric approaches can be computationally complex, whereas parametric approaches can be impacted by parameter sensitivity. A decoupled active contour (DAC) overcomes these problems by decoupling the external and internal energies and optimizing them separately. However a drawback of this approach is its reliance on the edge gradient as the external energy. This can lead to poor convergence toward the object boundary in the presence of weak object and strong background edges. To overcome these issues with convergence, a novel approach is proposed that takes advantage of a sparse texture model, which explicitly considers texture for boundary detection. The approach then defines the external energy as a weighted combination of textural and structural variation maps and feeds it into a multifunctional hidden Markov model for more robust object boundary detection. The enhanced DAC (EDAC) is qualitatively and visually analyzed on two natural image data sets as well as Brodatz images. The results demonstrate that EDAC effectively combines texture and structural information to extract the object boundary without impact on computation time and a reliance on color.

A novel statistical textural distinctiveness approach for robustly detecting salient regions in natural images is proposed. Rotational-invariant neighborhood-based textural representations are extracted and used to learn a set of representative texture atoms for defining a sparse texture model for the image. Based on the learnt sparse texture model, a weighted graphical model is constructed to characterize the statistical textural distinctiveness between all representative texture atom pairs. Finally, the saliency of each pixel in the image is computed based on the probability of occurrence of the representative texture atoms, their respective statistical textural distinctiveness based on the constructed graphical model, and general visual attentive constraints. Experimental results using a public natural image dataset and a variety of performance evaluation metrics show that the proposed approach provides interesting and promising results when compared to existing saliency detection methods.

A sparse texture dictionary learning method for unsupervised land cover classification is presented. The method takes the stance that land cover in remote sensing data is best analysed in texture patches rather than localized pixels. To this end, a feature vector is designed that describes local texture information in a spatially coherent manner. This texture model is extracted for each pixel in the scene. A sparse dictionary of global texture models is then learned to characterize the underlying texture distribution of the scene in a simplified manner. An unsupervised classifier is learned using these global texture models for grouping pixels exhibiting high similarity. Being an unsupervised classifier, the class labels that are learned are unbiased toward human interpretation of the scene, and rather are learned according to the texture information. The method is validated using polarimetric SAR data over a Flevoland, Netherlands agriculture scene, but may be generalized to any remote sensing data. Promising experimental results show how the proposed method retains the spatial coherence of crops, and attains higher accuracy than recent unsupervised and supervised classification methods using the same data.

A texture sparsifying transform for use in unsupervised classification of sea-ice in polarimetric synthetic aperture radar (SAR) imagery is presented. The goal of the sparsifying transform is to compactly represent the underlying information of the SAR imagery to eliminate sources of unwanted noise and complexities (e.g., banding effect on RADARSAT-2) commonly found in SAR imagery. The proposed algorithm is designed to be simple to implement and discriminative in sea-ice scenes. Performing unsupervised classification on the sparsifying transform space using scenes captured with C-band HV polarization yields experimental results that are much more accurate than common pixel-based methods, and performs comparably to a recent more complex method.

A novel model is presented to address the problem of semantic clustering of geo-objects in very high resolution panchromatic satellite images. The proposed model combines a probabilistic topic model with a multiscale image representation into an automatic framework by embedding both document and scale selections. The probabilistic topic model is used to characterize the statistical distributions of both intraclass appearance and inter-class coherence of geo-objects within documents, i.e., squared sub-images. Because the bag-of-words assumption involved in the probabilistic topic models does not consider the spatial coherence between topic labels, the multiscale image representation is designed to provide a self-adaptive spatial regularization for various geo-object categories. By introducing scale and document selections, the automatic framework integrates the probabilistic topic model and the multiscale image representation to ensure that words on a site should be allocated the same topic label no matter what documents they reside in. Consequently, unlike the traditional method of applying topic models for analyzing satellite images, the process of explicitly generating a set of documents before modeling and then combining multiple labels for a word on a given site is unnecessary. Gibbs sampling is adopted for parameter estimation and image clustering. Extensive experimental evaluations are designed to first analyze the effect of parameters in the proposed model and then compare the results of our model with those of some state-of-the-art methods for three different types of images. The results indicate that the proposed algorithm consistently outperforms these exiting state-of-the-art methods in all of the experiments.

Despite continuous improvements in optical flow in the last three decades, the ability for optical flow algorithms to handle illumination variation is still an unsolved challenge. To improve the ability to interpret apparent object motion in video containing illumination variation, an illumination-robust optical flow method is designed. This method decouples brightness into reflectance and illumination components using a stochastic technique; reflectance is given higher weight to ensure robustness against illumination, which is suppressed. Illumination experiments using the Middlebury and University of Oulu databases demonstrate the decoupled method's improvement when compared with state-of-the-art. In addition, a novel technique is implemented to visualize optical flow output, which is especially useful to compare different optical flow methods in the absence of the ground truth.

Two major challenges faced in active contours are poor capture range and high sensitivity towards noise. Recently, the concept of tensor vector convolution (TVF) was introduced and shown to be promising in handling these challenges. However, in the presence of high noise levels, TVF may have difficulty in converging to the desired object boundary, particularly if the distance is great between the initial contour and the object boundary. To tackle this challenge, the concept of a multi-scale tensor vector field (MTVF) active contour is introduced to further reduce noise sensitivity. Comparing the performance of MTVF with multi-scale gradient vector field and multi-scale vector field convolution demonstrates that MTVF is more resilient to high noise levels as well as significantly reducing computation time.

Thousands of spaceborne synthetic aperture radar (SAR) sea-ice images are systematically processed every year in support of operational activities such as ship navigation and environmental monitoring. An automated approach that generates pixel-level sea-ice image classification is required since manual pixel-level classification is not feasible. Currently, using a standardized approach, trained ice analysts manually segment full SAR scenes into smaller polygons to record ice types and concentrations. Using these data, pixel-level classification can be achieved by initial unsupervised segmentation of each polygon, followed by automatic sea-ice labeling of the full scene. A fully automated Markov random field model that is used to assign labels to all segmented regions in the full scene has been designed and implemented. This approach is the first known successful end-to-end process for operational SAR sea-ice image classification. In addition, a novel performance evaluation framework has been developed to validate the segmentation and labeling of SAR sea-ice images. A trained sea-ice expert has conducted an arms length evaluation using this framework to generate a set of full-scene reference images used for testing. Testing demonstrates operational success of the labeling approach.

A region-based unsupervised segmentation and classification algorithm for polarimetric synthetic aperture radar (SAR) imagery that incorporates region growing and a Markov random field edge strength model is designed and implemented. This algorithm is an extension of the successful Iterative Region Growing with Semantics (IRGS) segmentation and classification algorithm, which was designed for amplitude only SAR imagery, to polarimetric data. Polarimetric IRGS (PolarIRGS) extends IRGS by incorporating a polarimetric feature model based on the Wishart distribution and modifying key steps such as initialization, edge strength computation, and the region growing criterion. Like IRGS, PolarIRGS oversegments an image into regions and employs iterative region growing to reduce the size of the solution search space. The incorporation of an edge penalty in the spatial context model improves segmentation performance by preserving segment boundaries that traditional spatial models will smooth over. Evaluation of PolarIRGS with Flevoland fully polarimetric data shows that it improves upon two other recently published techniques in terms of classification accuracy.

Spatio-temporal salient features can localize the local motion events and are used to represent video sequences for many computer vision tasks such as action recognition. The robust detection of these features under geometric variations such as affine transformation and view/scale changes is however an open problem. Existing methods use the same filter for both time and space and hence, perform an isotropic temporal filtering. A novel anisotropic temporal filter for better spatio-temporal feature detection is developed. The effect of symmetry and causality of the video filtering is investigated. Based on the positive results of precision and reproducibility tests, we propose the use of temporally asymmetric filtering for robust motion feature detection and action recognition.

Identifying the surfaces of three-dimensional static objects or of two-dimensional objects over time are key to a variety of applications throughout computer vision. Active surface techniques have been widely applied to such tasks, such that a deformable spline surface evolves by the influence of internal and external (typically opposing) energies until the model converges to the desired surface. Present deformable model surface extraction techniques are computationally expensive and are not able to reliably identify surfaces in the presence of noise, high curvature, or clutter. This paper proposes a novel active surface technique, decoupled active surfaces, with the specific objectives of robustness and computational efficiency. Motivated by recent results in two-dimensional object segmentation, the internal and external energies are treated separately, which leads to much faster convergence. A truncated maximum likelihood estimator is applied to generate a surface consistent with the measurements (external energy), and a Bayesian linear least squares estimator is asserted to enforce the prior (internal energy). To maintain tractability for typical three-dimensional problems, the density of vertices is dynamically resampled based on curvature, a novel quasi-random search is used as a substitute for the ML estimator, and sparse conjugate-gradient is used to execute the Bayesian estimator. The performance of the proposed method is presented using two natural and two synthetic image volumes.

Among the main limitations of active contours are their high noise sensitivity and poor capture range from the target object. One of the most promising approaches for addressing these limitations is the concept of Vector Field Convolution (VFC). However, due to its isotropic vector field kernel, VFC does not take full advantage of the underlying image structural characteristics. By specifically addressing this idea, a novel local tensor vector field approach is developed to adaptively account for these structural characteristics. Experimental results demonstrate that the proposed adaptive method leads to more accurate segmentation.

A novel nonlinear volumetric scale-space framework is proposed for multi-scale volumetric data representation. The problem is formulated as a Bayesian least-squares estimator, and a quasi-random density estimation approach is introduced for estimating the posterior distribution between consecutive volumetric scale space realizations. Experimental results using both synthetic and real MR volumetric data demonstrate the effectiveness of the proposed scale-space framework for three-dimensional representation with significantly better structural separation and localization across all scales when compared to existing volumetric scale-space frameworks such as volumetric anisotropic diffusion and volumetric linear Gaussian scale-space, especially under scenarios with high noise levels.

In this paper, the effects of employing different noise estimation strategies on the performance of noise artifact suppression techniques in achieving high image quality has been investigated. Most literature on the subject tends to use the true noise level of the noisy image when performing noise artifact suppression. However, this approach does not reflect how such techniques would be used in practical situations where the true noise level is unknown, which is common in most image and video processing applications. Therefore, in practical situations, the noise level must first be estimated before a noise artifact suppression technique can be applied using the estimated noise level. Through a comprehensive analysis of different noise estimation strategies using empirical testing on a variety of images with different characteristics, the MAD wavelet noise estimation technique was found to be the overall preferred noise estimation technique for all popular noise artifact suppression techniques investigated (BM3D, bilateral, Neigh Shrink, BLS-GSM and non-local means). Furthermore, the BM3D noise artifact suppression technique, combined with the MAD wavelet noise estimation technique, was found to offer the best performance in achieving high image quality in situations where the noise level is unknown and must be estimated. The outcome of this research is clear recommendations that can be used in practise when suppressing noise artifacts exhibited in digital imagery and video.

A novel energy functional for automatic registration of remote sensing imagery based on quasi-random scale-space structural correlation is presented. The structural correlation energy functional takes advantage of the fact that, for many types of remote sensing imagery, there exist common structures at different scales even if the acquired images have very different intensity characteristics. The proposed energy functional also takes advantage of the noise robustness and feature localization properties of quasi-random scale-space theory. An efficient globally exhaustive optimization strategy in the frequency domain is developed for registering remote sensing imagery based on the proposed energy functional. Promising test results on interband, intraband, and intermodal remote sensing image sets show that the proposed method has the advantage of being robust to differing sensing conditions and large misalignments.

The accurate detection of object boundaries via active contours is an ongoing research topic in computer vision. Most active contours converge toward some desired contour by minimizing a sum of internal (prior) and external (image measurement) energy terms. Such an approach is elegant, but suffers from a slow convergence rate and frequently misconverges in the presence of noise or complex contours. To address these limitations, a decoupled active contour (DAC) is developed which applies the two energy terms separately. Essentially, the DAC consists of a measurement update step, employing a Hidden Markov Model (HMM) and Viterbi search, and then a separate prior step, which modifies the updated curve based on the relative strengths of the measurement uncertainty and the nonstationary prior. By separating the measurement and prior steps, the algorithm is less likely to misconverge; furthermore, the use of a Viterbi optimizer allows the method to converge far more rapidly than energy-based iterative solvers. The results clearly demonstrate that the proposed approach is robust to noise, can capture regions of very high curvature, and exhibits limited dependence on contour initialization or parameter settings. Compared to five other published methods and across many image sets, the DAC is found to be faster with better or comparable segmentation accuracy.

A novel method for synthesizing synthetic aperture radar (SAR) sea-ice imagery named IceSynth II is presented. A Markov random field model is assumed, and a conditional sampling approach is used to learn local conditional posterior probability distributions on a regional basis. Synthetic SAR sea-ice images and the associated ground-truth segmentations are generated using a region-based posterior sampling approach. Experimental results using single-polarization RADARSAT-1 and dual-polarization RADARSAT-2 SAR sea-ice imagery provided by the Canadian Ice Service show that IceSynth II is capable of producing SAR sea-ice imagery that is more realistic than existing approaches. The synthesized images are well suited for performing systematic and reliable objective evaluation of SAR sea-ice image segmentation methods.

Multivariate image segmentation is a challenging task, influenced by large intraclass variation that reduces class distinguishability as well as increased feature space sparseness and solution space complexity that impose computational cost and degrade algorithmic robustness. To deal with these problems, a Markov random field (MRF) based multivariate segmentation algorithm called "multivariate iterative region growing using semantics" (MIRGS) is presented. In MIRGS, the impact of intraclass variation and computational cost are reduced using the MRF spatial context model incorporated with adaptive edge penalty and applied to regions. Semantic region growing starting from watershed over-segmentation and performed alternatively with segmentation gradually reduces the solution space size, which improves segmentation effectiveness. As a multivariate iterative algorithm, MIRGS is highly sensitive to initial conditions. To suppress initialization sensitivity, it employs a region-level k -means (RKM) based initialization method, which consistently provides accurate initial conditions at low computational cost. Experiments show the superiority of RKM relative to two commonly used initialization methods. Segmentation tests on a variety of synthetic and natural multivariate images demonstrate that MIRGS consistently outperforms three other published algorithms.

In this paper, we propose a novel edge-preserving region (EPR)-based representation for synthetic aperture radar (SAR) images, which is incorporated with a region-level Markov random field (MRF) model to offer an efficient approach to the segmentation of SAR sea ice images. The EPR-based representations of SAR images are constructed by applying the speckle reduction anisotropic diffusion (SRAD) algorithm and the watershed transform, which aims at suppressing oversegmentation within objects while accurately locating object edges at region boundaries in the presence of speckle noise. In combination with a region-level MRF, the EPR-based representation largely reduces the search space of optimization process and improves parameter estimation of feature model, leading to considerable computational savings and less probability of false segmentation. Relative to the existing region-level MRF-based methods, testing results have demonstrated that the proposed method achieves more than 50% reduction of computational time and improves the segmentation accuracy especially at high speckle noise.

This paper proposes a structure-preserving speckle reduction (SPSR) algorithm for synthetic aperture radar (SAR) images by exploiting self-similarity of structural patterns based on nonlocal means filter. The SPSR algorithm is featured by discerning pixels of similar structural patterns, which is crucial for a despeckling process to avoid blurring image structure. To alleviate the impact of speckle noise to similarity measure, a two-stage filtering scheme is introduced into the SPSR algorithm. Filtering at the first stage aims at an accurate approximation of true structural similarity, followed by the filtering at the second stage to group pixels with similar neighborhood in a large area. Compared to the traditional Lee filter, enhanced Lee filter and the speckle reducing anisotropic diffusion (SRAD), evaluation results have shown that the SPSR algorithm substantially improves the despeckling performance especially on structure preservation and speckle reduction in homogeneous regions.

The benefits of augmenting Advanced Microwave Scanning Radiometer for the Earth Observing System (AMSR-E) image data with Quick Scatterometer (QuikSCAT) image data for supervised sea ice classification in the Western Arctic region are investigated. Experiments compared the performance of a maximum likelihood classifier when used with the AMSR-E-only data set against using the combined data. The preferred number of bands to use for classification was examined, as well as whether principal component analysis (PCA) can be used to reduce the dimensionality of the data. The reliability of training data over time was also investigated. Adding QuikSCAT often improves classifier accuracy in a statistically significant manner and never decreases it significantly when a sufficient number of bands are used. Combining these data sets is beneficial for sea ice mapping. Using all available bands is recommended, data fusion with PCA does not offer any benefit for these data, and training data from a specific date remains reliable within 30 days.

The extraction of contours using deformable models, such as snakes, is a problem of great interest in computer vision, particular in areas of medical imaging and tracking. Snakes have been widely studied and many methods are available. In most cases, the snake converges towards the optimal contour by minimizing a sum of internal (prior) and external (image measurement) energy terms. This approach is elegant, but frequently mis-converges in the presence of noise or complex contours. To address these limitations, a novel discrete snake is proposed which treats the two energy terms separately. Essentially, the proposed method is a deterministic iterative statistical data fusion approach, in which the visual boundaries of the object are extracted, ignoring any prior, employing a Hidden Markov Model (HMM) and Viterbi search, and then applying importance sampling to the boundary points, on which the shape prior is asserted. The proposed implementation is straightforward and achieves dramatic speed and accuracy improvement compared to other methods.

This paper introduces a novel perceptually adaptive approach to image denoising using anisotropic non-local means. In the classical non-local means image denoising approach, the value of a pixel is determined based on the weighted average of other pixels, where the weights are determined based on a fixed isotropically weighted similarity function between the local neighborhoods. In the proposed algorithm, we demonstrate that noticeably improved perceptual quality can be achieved through the use of adaptive anisotropically weighted similarity functions between local neighborhoods. This is accomplished by adapting the similarity weighing function in an anisotropic manner based on the perceptual characteristics of the underlying image content derived efficiently based on the Mexican Hat wavelet. Experimental results show that the proposed method can be used to provide improved perceptual quality in the denoised image both quantitatively and qualitatively when compared to existing methods.

This paper presents a novel method for medical image signal fusion using complex-valued wavelets to enhance the information content in the fused signal from a perceptual manner. The proposed method introduces an adaptively weighted aggregation of signal characteristics based on the phase characteristics of medical image signals. The proposed method exploits the phase characteristics of the image signals to adaptively accentuate important anatomical and functional characteristics captured by each image signal during the signal fusion process. Experimental results show that the proposed method can improve the visualization of important anatomical and functional characteristics from different medical image signals in the fused image signal when compared with non-adaptive image signal fusion methods.

This paper introduces a novel stochastic approach to image denoising using an adaptive Monte Carlo scheme. Random samples are generated from the image field using a spatially-adaptive importance sampling approach. Samples are then represented using Gaussian probability distributions and a sample rejection scheme is performed based on a chi2statistical hypothesis test. The remaining samples are then aggregated based on Pearson Type VII statistics to create a non-linear estimate of the denoised image. The proposed method exploits global information redundancy to suppress noise in an image. Experimental results show that the proposed method provides superior noise suppression performance both quantitatively and qualitatively when compared to the state-of-the-art image denoising methods.

This paper proposes an image segmentation method named iterative region growing using semantics (IRGS), which is characterized by two aspects. First, it uses graduated increased edge penalty (GIEP) functions within the traditional Markov random field (MRF) context model in formulating the objective functions. Second, IRGS uses a region growing technique in searching for the solutions to these objective functions. The proposed IRGS is an improvement over traditional MRF based approaches in that the edge strength information is utilized and a more stable estimation of model parameters is achieved. Moreover, the IRGS method provides the possibility of building a hierarchical representation of the image content, and allows various region features and even domain knowledge to be incorporated in the segmentation process. The algorithm has been successfully tested on several artificial images and synthetic aperture radar (SAR) images.

The papers in this special section are devoted to pattern recognition in remote sensing applications.

This paper presents the Automatic Registration of Remote-Sensing Images (ARRSI); an automatic registration system built to register satellite and aerial remotely sensed images. The system is designed specifically to address the problems associated with the registration of remotely sensed images obtained at different times and/or from different sensors. The ARRSI system is capable of handling remotely sensed images geometrically distorted by various transformations such as translation, rotation, and shear. Global and local contrast issues associated with remotely sensed images are addressed in ARRSI using control-point detection and matching processes based on a phase-congruency model. Intensity-difference issues associated with multimodal registration of remotely sensed images are addressed in ARRSI through the use of features that are invariant to intensity mappings during the control-point matching process. An adaptive control-point matching scheme is employed in ARRSI to reduce the performance issues associated with the registration of large remotely sensed images. Finally, a variation on the Random Sample and Consensus algorithm called Maximum Distance Sample Consensus is introduced in ARRSI to improve the accuracy of the transformation model between two remotely sensed images while minimizing computational overhead. The ARRSI system has been tested using various satellite and aerial remotely sensed images and evaluated based on its accuracy and computational performance. The results indicate that the registration accuracy of ARRSI is comparable to that produced by a human expert and improvement over the baseline and multimodal sum of squared differences registration techniques tested.

Synthetic aperture radar (SAR) has been intensively used for sea-ice monitoring in polar regions. A computer-assisted analysis of SAR sea-ice imagery is extremely difficult due to numerous imaging parameters and environmental factors. This paper presents a system which, with some limited information provided, is able to perform an automated segmentation and classification for the SAR sea-ice imagery. In the system, both the segmentation and classification processes are based on a Markov random-field theory and are formulated in a joint manner under the Bayesian framework. Solutions to the formulation are obtained by a region-growing technique which keeps refining the segmentation and producing semantic class labels at the same time in an iterative manner. The algorithm is a general-segmentation approach named iterative region growing using semantics, which, in this paper, is dedicated to the problem of classifying the operational SAR sea-ice imagery provided by the Canadian Ice Service (CIS). The classified image results have been validated by the CIS personnel, and the resulting classifications are quite successful using the same algorithm applied to diverse data sets.

Modeling spatial context constraints using a Markov random field (MRF) has been widely used in the segmentation of noisy images. Its applicability to synthetic aperture radar (SAR) sea-ice segmentation has also been demonstrated recently. However, most existing MRF models are not capable of preserving filaments, specifically leads and ridges for SAR sea ice, which are valuable for ship navigation applications and necessary for identifying certain ice types. In this paper, a new statistical context model is proposed that, within the same scene, can simultaneously preserve narrow elongated features while producing similar smooth segmentation results comparable to typical MRF-based approaches. Tested on one synthetic image and two SAR sea-ice scenes, this filament preserving model substantially improves classification accuracies when compared to standard Gaussian mixture and MRF-based segmentation algorithms

Research was conducted to develop a methodology to model the emotional content of music as a function of time and musical features. Emotion is quantified using the dimensions valence and arousal, and system-identification techniques are used to create the models. Results demonstrate that system identification provides a means to generalize the emotional content for a genre of music. The average R2 statistic of a valid linear model structure is 21.9% for valence and 78.4% for arousal. The proposed method of constructing models of emotional content generalizes previous time-series models and removes ambiguity from classifiers of emotion.

Environmental and sensor challenges pose difficulties for the development of computer-assisted algorithms to segment synthetic aperture radar (SAR) sea ice imagery. In this research, in support of operational activities at the Canadian Ice Service, images containing visually separable classes of either ice and water or multiple ice classes are segmented. This work uses image intensity to discriminate ice from water and uses texture features to identify distinct ice types. In order to seamlessly combine image spatial relationships with various image features, a novel Bayesian segmentation approach is developed and applied. This new approach uses a function-based parameter to weight the two components in a Markov random field (MRF) model. The devised model allows for automatic estimation of MRF model parameters to produce accurate unsupervised segmentation results. Experiments demonstrate that the proposed algorithm is able to successfully segment various SAR sea ice images and achieve improvement over existing published methods including the standard MRF-based method, finite Gamma mixture model, and K-means clustering.

This paper presents a map-guided sea ice classification system built to work in parallel with the Canadian Ice Service (CIS) operations to produce pixel-based ice maps that complement actual "egg code" maps produced by CIS. The system uses the CIS maps as input to guide classification by providing information on the number of ice types and their final label for specific regions. Segmentation is based on a modified adaptive Markov random field (MRF) model that uses synthetic aperture radar (SAR) intensities and texture features as input. The ice type labeling is performed automatically by gathering evidences based on a priori information on one or two classes and deducing the other labels iteratively by comparing distributions of segments. Three methods for comparing the segment distributions (Fisher criterion, Mahalanobis distance, and Kolmogorov-Smirnov test) were implemented and compared. The system is fully described with special attention to the labeling procedure. Examples are presented in the form of two CIS SAR-based ice maps from the Gulf of Saint Lawrence region and one example from the Beaufort Sea. The results indicate that when the segmentation is good, the labeling attains best results (between 71% and 89%) based on evaluation by a sea ice analyst. Some problems remain to be assessed which are primarily attributable to discrepancies in the information provided by the egg code and what is actually visible in the SAR image. Subscale information on floe size and shape available to human analysts, but not in this classification system, also appear to be a critical information for separating some ice types.

A design-based method to fuse Gabor filter and grey level co-occurrence probability (GLCP) features for improved texture recognition is presented. The fused feature set utilizes both the Gabor filter's capability of accurately capturing lower and mid-frequency texture information and the GLCP's capability in texture information relevant to higher frequency components. Evaluation methods include comparing feature space separability and comparing image segmentation classification rates. The fused feature sets are demonstrated to produce higher feature space separations, as well as higher segmentation accuracies relative to the individual feature sets. Fused feature sets also outperform individual feature sets for noisy images, across different noise magnitudes. The curse of dimensionality is demonstrated not to affect segmentation using the proposed the 48-dimensional fused feature set. Gabor magnitude responses produce higher segmentation accuracies than linearly normalized Gabor magnitude responses. Feature reduction using principal component analysis is acceptable for maintaining the segmentation performance, but feature reduction using the feature contrast method dramatically reduced the segmentation accuracy. Overall, the designed fused feature set is advocated as a means for improving texture segmentation performance.

Features based on Markov random field (MRF) models are sensitive to texture rotation. This paper develops an anisotropic circular Gaussian MRF (ACGMRF) model for retrieving rotation-invariant texture features. To overcome the singularity problem of the least squares estimate method, an approximate least squares estimate method is designed and implemented. Rotation-invariant features are obtained from the ACGMRF model parameters using the discrete Fourier transform. The ACGMRF model is demonstrated to be a statistical improvement over three published methods. The three methods include a Laplacian pyramid, an isotropic circular GMRF (ICGMRF), and gray level cooccurrence probability features.

This paper compares the discrimination ability of two texture analysis methods: Markov random fields (MRFs) and gray-level cooccurrence probabilities (GLCPs). There exists limited published research comparing different texture methods, especially with regard to segmenting remotely sensed imagery. The role of window size in texture feature consistency and separability as well as the role in handling of multiple textures within a window are investigated. Necessary testing is performed on samples of synthetic (MRF generated), Brodatz, and synthetic aperture radar (SAR) sea ice imagery. GLCPs are demonstrated to have improved discrimination ability relative to MRFs with decreasing window size, which is important when performing image segmentation. On the other hand, GLCPs are more sensitive to texture boundary confusion than MRFs given their respective segmentation procedures.

The features based on Markov random field (MRF) models are usually sensitive to the rotation of image textures. The paper develops an anisotropic circular Gaussian MRF (ACGMRF) model for modeling rotated image textures and retrieving rotation-invariant texture features. To overcome the singularity problem of the least squares estimate (LSE) method, an approximate least squares estimate (ALSE) method is proposed to estimate the parameters of ACGMRF model. The rotation-invariant features can be obtained from the parameters of the ACGMRF model by the one-dimensional (1D) discrete Fourier transform (DFT). Significantly improved accuracy can be achieved by applying the rotation-invariant features to classify SAR (synthetic aperture radar) sea ice and Brodatz imagery.

Sea ice concentrations obtained with two algorithms from Special Sensor Microwave/Imager (SSM/I) data are compared to spaceborne visible/infrared and active microwave imagery for the Greenland Sea in spring. Both algorithms, the ARTIST Sea Ice algorithm (ASI) and the SEA LION algorithm (SLA), utilize 85-GHz SSM/I brightness temperatures with a spatial resolution of 15 km /spl times/13 km. Ice concentrations obtained from Advanced Very High Resolution Radiometer (AVHRR) infrared data in cloud-free areas are underestimated by SLA and ASI ice concentrations by 3.6% and 8.3% (correlation coefficients of 0.90 and 0.91). Ice concentrations estimated from texture classified ERS-2 synthetic aperture radar (SAR) images by assigning experience-based ice concentrations to ice-type classes are overestimated by SLA and ASI ice concentrations by 4.4% and 1.5% (correlation coefficients of 0.84 and 0.77). However, omitting low/high ice concentrations forming up to 80% (AVHRR) and 60% (SAR) of the entire dataset reveals a significantly different statistic. For instance, the correlation between AVHRR and SLA and ASI ice concentrations drops to 0.77 and 0.70, respectively. All presented techniques to obtain ice concentrations need improvement and future developments should involve larger datasets. However, with care, both algorithms can be used to obtain reasonable ice concentration maps with a 12.5 km /spl times/12.5 km grid-cell size.

Image registration is one of the basic image processing operations in remote sensing. With an increasing number of images collected every day from different sensors, automated registration of multi-sensor/multi-spectral images has become an important issue. A wide range of registration techniques exists for different types of applications and data sources, however no algorithm is known that can accurately register multi-source images consistently. This research addresses this problem by investigating the development of a fully automatic registration system for synthetic aperture radar (SAR) and optical remote sensing images. The development of this new automatic image registration method is based on the extraction and matching of common features that are visible in both images. The algorithm involves the following five steps: noise removal, edge extraction, edge linking pattern extraction and pattern matching. The application of the developed automatic image registration model to SAR and optical image pairs showed that accurate ground control points (GCPs) could be identified automatically.

A novel approach to geometric shape preservation of remote sensing SAR sea ice images is presented in this paper. This approach will complement the existing edge detection schemes. The final edge is evolved by iterations. During each iteration all existing edges are checked for missing link and missing links are found by checking the orientation of the existing edges and by searching for best gradient path. When open edges are reduced considerably, opening of the features that are critical for shape preservation are found by tracing the edges and closing the boundary.

A critical shortcoming of determining co-occurrence probability texture features using Haralick's popular grey level co-occurrence matrix (GLCM) is the excessive computational burden. Here, a more robust algorithm (the grey level cooccurrence integrated algorithm or GLCIA) to perform this task is presented. The GLCIA is created by integrating the preferred aspects of two algorithms: the grey level cooccurrence hybrid structure (GLCHS) and the grey level cooccurrence hybrid histogram (GLCHH). The GLCHS utilizes a dedicated 2-d data structure to quickly generate the probabilities and apply statistics to generate the features. The GLCHH uses a more efficient 1-d data structure to perform the same tasks. Since the GLCHH is faster than the GLCHS yet the GLCHH is not able to calculate features using all available statistics, the integration of these two methods generates a superior algorithm (the GLCIA). The computational gains vary as a function of window size, quantization level, and statistics selected. The GLCIA computational time relative to that of the standard GLCM method ranges from 0.04% to 16%. The GLCIA is a highly recommended technique for anyone wishing to calculate co-occurrence probability texture features, especially from large-scale digital imagery.

Spatial fluctuations in microwave backscatter may be an important piece of information in discriminating tree stands. However, the presence of speckle in synthetic aperture radar (SAR) image data is a barrier to the exploitation of image texture. The authors explored a new methodology that combines a recent adaptive speckle reduction algorithm by Lopes et al. (1990) with a generic texture estimation scheme. They investigated the claim that this filter was capable of preserving backscatter texture. To understand if speckle reduction was destroying backscatter texture, they compared the strength of the relationship between forest inventory parameters and image texture as a function of spatial scale for both filtered and unfiltered images. They used Radarsat Fine mode image data: single look resolution is approximately 8.5 m, and pixel spacing is 3 m. Their study area was northern Vancouver Island, B.C., on the west coast of Canada. For the unfiltered data, they found that the ability of image texture to predict the forest parameters decreased as the texture scale increased from 3 to 13 m, suggesting greater information content in the small scale texture. For the filtered data, this relationship was much weaker at small scales and was not a function of distance. Their results suggest that the speckle filter was not retaining small scale texture, which is consistent with the theoretical hypotheses underlying its multiplicative noise model. They also show that there is significant information in small state SAR image texture that may be used as an adjunct to other spatial information for discriminating tree stands in the temperate rain forest.

A critical shortcoming of determining texture features derived from grey-level co-occurrence matrices (GLCM's) is the excessive computational burden. This paper describes the implementation of a linked-list algorithm to determine co-occurrence texture features far more efficiently. Behavior of common co-occurrence texture features across difference grey-level quantizations is investigated.