Single underwater image often suffer from severe quality degradation and field-of-view limitation due to the underwater light propagation characteristics and the viewing range of camera equipment. To address these challenges, we propose a underwater scene clarity reconstruction framework called USCR, which comprises a multilayer information fusion (MIF) method for underwater image enhancement (UIE) and a self-organized stitching (SOS) method for image stitching. First, MIF corrects color distortion, enhances contrast, and highlights image detail information through a minimally attenuated channel guided color correction strategy and a gradient weight fusion strategy. Subsequently, SOS is applied to stitch the enhanced underwater images, which utilizes a homography matrix to initially stitch the image sequence, and further employs a pixel blending strategy based on boundary distance weighting for boundary pixel fusion to the initial stitch image, aiming to ensure a homogeneous transition of the stitch region. Our reconstructed underwater scenes are characterized by visual clarity and a wide field-of-view. Extensive qualitative and quantitative experimental validations show that USCR outperforms the state-of-the-art methods in underwater visual reconstruction task.

This paper proposes a family of Aligned Entropic Graph Kernels (AEGK) for graph classification, based on the Averaged Mixing Matrix (AMM) of Continuous-time Quantum Walks (CTQWs). Specifically, we show how the AMM matrix allows us to compute a quantum Shannon entropy of each vertex for either un-attributed or attributed graphs. For pairwise graphs, the proposed AEGK kernels are defined by computing the kernel-based similarity between the quantum Shannon entropies of their pairwise aligned vertices. Theoretical analysis reveals that the AEGK kernels can not only integrate the structural correspondence information between graphs, but also discriminate the structural differences between aligned vertices. Moreover, the AEGK kernels can simultaneously capture both global and local structural characteristics through the quantum Shannon entropies. These theoretical properties explain the effectiveness.

Semantic segmentation is commonly used for Oil Spill Detection (OSD) in remote sensing images. However, the limited availability of labelled oil spill samples and class imbalance present significant challenges that can reduce detection accuracy. Furthermore, most existing methods, which rely on convolutional neural networks (CNNs), struggle to detect small oil spill areas due to their limited receptive fields and inability to effectively capture global contextual information. This study explores the potential of State-Space Models (SSMs), particularly Mamba, to overcome these limitations, building on their recent success in vision applications. We propose OSDMamba, the first Mambabased architecture specifically designed for oil spill detection. OSDMamba leverages Mamba’s selective scanning mechanism to effectively expand the model’s receptive field while preserving critical details. Moreover, we designed an asymmetric decoder incorporating ConvSSM and deep supervision to strengthen multiscale feature fusion, thereby enhancing the model’s sensitivity to minority class samples. Experimental results show that the proposed OSDMamba achieves state-of-the-art performance, yielding improvements of 8.9% and 11.8% in OSD across two publicly available datasets. The source codes will be made publicly available at https://github.com/Chenshuaiyu1120/Oil-Spill-detection.

Confucius remarked, “When three men meet together, one of them who is anxious to learn can always learn something of the other two. He can profit by the good example of the one and avoid the bad example of the other”. 1 In the light of these remarks, we develop a Confucius tri-learning paradigm of learning from both good examples and bad examples. Specifically, we propose to train three models, i.e., two classifiers and one generator, together. On the one hand, each of the two classifiers can learn from “good” examples provided by the other in a recyclable manner. This reduces the amount of incorrect quasi-labels in training cycles, and thus enables a comprehensive use of unlabeled data to effectively train classifiers. On the other hand, each of the two classifiers can learn from “bad” examples given by the generator. By avoiding the bad examples, the negative impact of the incorrect quasi-labels is further neutralized such that refined classification results are obtained. These advantages are profitable for classification task in the condition that extremely limited labeled data samples are available for training, because the “good” examples augment the labeled data samples for training and the “bad” examples lift the classifiers’ discrimination ability against fake targets. The experiments on the MSTAR, OpenSARShip, and FUSAR-Ship datasets demonstrate that our paradigm gives state-of-the-art results. We release our implementation source code at https://gitee.com/han-zongjun/confucius-tri-learning for public evaluations.

Underwater image captioning bridges the gap between visual perception and semantic understanding of underwater scenes, playing a crucial role in applications such as ocean geoscience and underwater remote sensing. Despite progress in this field, limitations remain in achieving accurate underwater image captioning. The main limitations are: (a) the underestimation of basic sketch features in underwater image captioning, and (b) insufficient consideration of the impact of scale differences in underwater objects. To overcome these limitations, we propose underwater image captioning with AquaSketch enhanced cross-scale information fusion. Our novel contributions are twofold: (a) A novel AquaSketch (i.e., aqua sketch) enhancement method is developed to reduce the impact of underwater image distortion on scene understanding, while enhancing both detailed and background information; and (b) A top-down dual-branch pyramid for cross-scale information fusion is proposed. This architecture fuses multi-scale feature information from two branches through an attention-based feature fusion structure, performing cross-scale fusion in a top-down manner. The resulting pyramid fusion features offer a comprehensive representation of underwater object information. Collectively, these contributions facilitate the generation of accurate and comprehensive underwater image captions. Experimental evaluations on three datasets demonstrate that our proposed underwater image captioning model achieves state-of-the-art performance in the field.

Unsupervised domain adaptation (UDA) aims to improve model performance in the target domain by leveraging labeled data from the source domain while not requiring labeled data in the target domain. It has been widely applied in cross-domain semantic segmentation of remote sensing images (RSIs). Despite some advancements in this area, challenges such as class confusion due to color and texture similarities, class imbalance due to significant scale variations and sample imbalance continue to impede progress in UDA for RSI segmentation. To address these challenges, we propose a novel self-supervised teacher-student network framework, including two innovative techniques: mask-enhanced class mix (MECM) and scale-based rare class sampling (SRCS). The MECM method applies a high proportion of masks to mixed images derived from both source-domain images and target-domain images, which encourages the model to infer the semantic information of masked areas from the surrounding context, enhancing cross-domain contextual semantic learning and improving the recognition accuracy of similar classes. Additionally, SRCS increases the sampling proportion of small-scale rare classes, mitigating the issue of class imbalance. Experiments show that our method outperforms existing UDA techniques in terms of PA, mF1, and mIoU, achieving state-of-the-art results on three public datasets. Notably, in the Potsdam IRRG to Vaihingen UDA scenario, our method’s performance on the key metric, mIoU, even surpasses that of supervised training, demonstrating the superiority of our approach. Codes are available athttps://github.com/Qiuyb-ai/UDA-With-ME-and-BS.

We develop a deep cross-view hashing method for street view image geo-localization by referencing remote sensing images with location information. We commence by exploiting attention-weighted image feature extractor models to construct feature descriptors for both a street view image (as a query) and a set of remote sensing images (as a database) with Universal Transverse Mercator (UTM) information. The cross-view similarity between street view and remote sensing feature descriptors for a common location is effectively maintained by the Vision Transformer (ViT) backbone in the models. We then design hash encoders to convert the feature descriptors into hash codes. By retrieving remote sensing images in terms of small Hamming distance values between the cross-view hash codes, we achieve preliminary geo-localization of the query street view image. This hashing technique enables fast localization with economical computational costs. We further refine the preliminary geo-localization results by applying a geographically clustered averaging method to the locations of the retrieved remote sensing images, resulting in exact geo-localization of the query street view image. The experimental results show that our method achieves fast geo-location of street view images on the CVACT [34] dataset. In scenes with a smaller field of view (FoV), its accuracy is not only on par with the existing state-of-the-art methods, but also demonstrates significant efficiency advantages.

GNSS-R technology provides a novel solution for sea ice dynamic monitoring through its all-weather observation capability and low-cost advantages. However, existing methods still face significant challenges in simultaneously achieving high-accuracy detection of ice-water boundaries and precise identification of large-area sea ice distant from transition zones. To address this issue, this paper proposes a novel sea ice detection model incorporating the semi-correlated supervision algorithm of dual neural networks. First, a dual-modal dataset comprising DDM and Differential DDM data is employed to capture the global scattering characteristics of sea ice and local differential features of transition regions, respectively. Second, an innovative dual-branch detection architecture is designed: the Differential DDM detection branch based on YOLOv11 achieves high-sensitivity identification of ice-water boundaries through local feature extraction, while the Efficient Vision Mamba detection branch enhances detection robustness for open water and continuous ice cover via global feature analysis. To further optimize model performance, a semi-correlated supervision correction algorithm is proposed, which dynamically integrates dual-branch detection results through temporal context analysis and bidirectional validation mechanisms, effectively resolving the performance imbalance between transition zones and stable regions in traditional single-model approaches. Experiments using J1-01B and TDS-1 satellites GNSS-R data demonstrate that the model achieves 94.1% accuracy for ice-water boundary detection, 97.3% accuracy for sea ice detection distant from transition zones, with an overall accuracy of 95.7%. This research establishes a high-accuracy, full-coverage technical framework for real-time polar sea ice monitoring, offering significant application value for global climate change studies and polar navigation safety.

Timely and accurate representation of sea surface dynamic fields is crucial for oil spill drift prediction. Numerically forecasted sea surface dynamic fields are available in a timely manner, but their accuracy is limited. Conversely, reanalysis sea surface dynamic fields offer superior accuracy but suffer from time delays. To enhance the performance of oil spill drift prediction, we propose a deep learning-based approach to correcting numerically forecasted sea surface dynamic fields, aligning them more closely with reanalysis sea surface dynamic fields. Our approach introduces an adversarial temporal convolutional network (ATCN) framework, consisting of a temporal convolutional network (TCN)-based corrector and a discriminator. The TCN can characterize sea surface dynamic field sequences both spatially and temporally. In this scenario, the corrector processes the numerically forecasted sea surface dynamic fields and outputs corrected sea surface dynamic fields that approximate the reanalysis sea surface dynamic fields. Adversarial training with the discriminator further refines the corrector. This approach enhances timely oil spill drift prediction using the corrected sea surface dynamic fields. We also provide a dataset of oil spill drifts from the Symphony and Sanchi accidents, including related sea surface dynamic field data and oil spill remote sensing data, establishing a baseline for evaluating oil spill drift prediction. Experiments on this dataset validate the ATCN framework’s effectiveness in enhancing oil spill drift prediction.

The underwater color disparity is an important cue for enhancing an underwater image. Applying the underwater color disparity indiscriminately to the entire underwater image tends to give rise to foreground-background crosstalk with either excessive foreground or insufficient background enhancement. To address the discriminativeness between underwater color disparities in foreground and background regions, we develop a discriminative underwater image enhancement method empowered by large foundation model technology. We first utilize the Segment Anything Model to generate segmentation masks, dividing the underwater image into foreground and background regions. This enables accurate foreground-background separation. Then, we conduct adaptive color compensation and fusion to improve the color histogram similarity for foreground and background regions separately. This corrects color deviations and improves contrasts in a discriminative manner that avoids the foreground-background crosstalk. Finally, we propose high-frequency edge fusion to extract high-frequency components from both the original underwater image and the fused image, and then fuse these components to obtain the final enhanced image. This eliminates blurred details arising from the discriminative processing of foreground and background regions. Our method represents the pioneering application of large foundation model technology to empower underwater image enhancement. Experimental results indicate that our method outperforms nine state-of-the-art underwater image enhancement methods in visual quality, achieves superior results across five underwater image quality evaluation metrics on three underwater image datasets, and is beneficial for practical applications such as underwater feature matching. We release our code at https://gitee.com/wanghaoupc/UIE SAM.

Remote sensing data from real-world scenarios manifests a long-tailed distribution, with the continuous emergence of new classes over time. Nevertheless, the existing class-incremental remote sensing classification models neglect the above long-tailed distribution phenomenon, which seriously damages their overall superior performance. Meanwhile, long-tail class-incremental learning developed in other areas focuses only on the classifier decision boundary optimization of the tail-class, while neglecting the robustness of the feature backbone. The feature backbone trained on the base classes causes a serious significant distribution shift for the incremental classes owing to the distributional differences between base and incremental classes. To solve these issues, we propose a continually evolved feature and classifiers learning (CEF-CL) framework for long-tail class-incremental remote sensing scene classification. Specifically, tail-class data are scaled and grafted onto head-class data to diversify the semantic information of the tail-class leveraging the rich context of the head classes, which can improve the generalization of the feature backbone. And then, an adaptive multi-scale feature fusion (AMFF) module is proposed to couple feature maps of head and tail classes scale by scale for generating virtual tail-class features that deeply perceive head-class information, which can further enhance the reliability of classifier decision boundary optimization. Furthermore, examples from old classes are regarded as pseudo-tail classes to participate in incremental learning, which greatly alleviates catastrophic forgetting of old classes. Extensive experiments on two remote sensing benchmarks demonstrate the superiority of the proposed CEF-CL in comparison with existing class-incremental learning.

One of the main challenges facing hyperspectral image super-resolution is the complex high dimensional data processing. Mamba leverages its ability to model long-range dependencies of linear complexity to capture the global spatial and spectral information of high-dimensional data while maintaining linear complexity. However, its visual state space equation mainly focuses on the band dimension mapping of the image, while ignoring the modeling of the spatial dimension. To overcome this limitation, we develop a Mamba hyperspectral image super-resolution framework, which comprises three essential components. The first component, i.e., spatial Mamba sub-network, models the spatial dimensions of hyperspectral data. It captures long-range dependencies in the pixel space, thereby integrating global spatial information into the framework. The second component, i.e., spectral Mamba sub-network, serves to capture long-range spectral dependencies. The third component, i.e., reconstruction, generates hyperspectral images with rich spatial and spectral details through pixel interpolation. Our Mamba framework fully develops the potential of the Mamba model in hyperspectral image super-resolution, significantly enhancing the restoration quality and accuracy of hyperspectral images. Extensive experiments on the Houston and QUST-1 datasets show that our framework outperforms state-of-the-art methods in both quantitative metrics and visual quality across diverse scenarios. We release our source code at https://gitee.com/xu_yinghao/MambaHSISR for public evaluations.

Accurate mapping of oil concentrations is essential for effective response to oil spill emergencies. The complexity of microwave scattering over oil-contaminated sea surfaces poses substantial challenges for synthetic aperture radar (SAR) applications, primarily due to the limited understanding of non-Bragg scattering mechanisms. This knowledge gap restricts the development of robust retrieval algorithms for quantifying oil spill concentrations. To address this issue, a novel retrieval approach is proposed based on the co-polarization difference (PD), which is independent of non-Bragg scattering. The influence of oil on the sea surface is attributed to two dominant factors: suppression of short gravity-capillary waves and reduction in the effective dielectric constant. By analyzing SAR imagery of oil spills with varying concentrations, it is found that the damping effect of oil spills on small-scale waves can be predicted using the Marangoni damping model. Once the contribution of wave suppression to PD reduction is isolated, the residual PD variation is attributed to changes in the dielectric constant. Oil concentration is then retrieved by comparing the PD of each pixel within the contaminated area to a theoretical PD lookup table. The proposed method is validated using simulated SAR datasets representing different oil concentrations and subsequently applied to SAR data acquired during the Deepwater Horizon oil spill in the Gulf of Mexico.

In this work, we develop a family of Aligned Entropic Graph Kernels (AEGK) for graph classification. We commence by performing the Continuous-time Quantum Walk (CTQW) on each graph structure, and compute the Averaged Mixing Matrix (AMM) to describe how the CTQW visits all vertices from a starting vertex. More specifically, we show how this AMM matrix allows us to compute a quantum Shannon entropy of each vertex for either un-attributed or attributed graphs. For pairwise graphs, the proposed AEGK kernels are defined by computing the kernel-based similarity between the quantum Shannon entropies of their pairwise aligned vertices. The analysis of theoretical properties reveals that the proposed AEGK kernels cannot only address the shortcoming of neglecting the structural correspondence information between graphs arising in most existing R-convolution graph kernels, but also overcome the problems of neglecting the structural differences and vertex-attributed information arising in existing vertex-based matching kernels. Moreover, unlike most existing classical graph kernels that only focus on the global or local structural information of graphs, the proposed AEGK kernels can simultaneously capture both global and local structural characteristics through the quantum Shannon entropies, reflecting more precise kernel-based similarity measures between pairwise graphs. The above theoretical properties explain the effectiveness of the proposed AEGK kernels. Experimental evaluations demonstrate that the proposed kernels can outperform state-of-the-art graph kernels and deep learning models for graph classification.

Phase linking is the crucial step in distributed scatterer InSAR processing, which determines the quality of time-series interferometric phases. The weight matrix is the key measure of phase linking method, which controls the participation of each interferometric pair for the single-master phase linking. Existing methods don’t consider the impacts of temporal-changed polarimetric scattering characteristics, leading to large phase closure errors. Based on the polarimetric stationarity, we propose a novel scattering consistency weight measure. Combined with the coherence weight, a joint weight is generated to improve three general phase linking methods. The methods are validated with time-series Radarsat-2 PolSAR data over Kilauea Volcano, Hawaii. The results show that the proposed methods obtain higher temporal posterior coherence and better equivalent single-master (ESM) interferometric phase than traditional methods.

The joint classification of hyperspectral image (HSI) and light detection and ranging (LiDAR) data has attracted considerable attention in the field of remote sensing. Integrating the advantages of the two data sources can provide precise data support and analytical decision-making for remote-sensing applications. However, due to the inherent differences in properties and semantic information from heterogeneous data, most existing deep-learning methods suboptimally extract the characteristic features of both data sources while utilizing their interactive information. In this letter, we propose a dual-branch feature fusion network-based cross-modal enhanced CNN and Transformer (DF2NCECT) to make full use of the respective features and interactive information of multisource data. DF2NCECT consists of two main stages. One is the basic feature extraction stage, which builds a hybrid convolution module based on 3DCNN and inception structure to fully extract the joint features of HSI from multiple spatial perspectives. The other is the deep feature fusion stage, where the CNN and Transformer are designed in parallel to fully explore and fuse deep features between HSI and LiDAR. More importantly, to achieve efficacious interactive information between HSI and LiDAR, a cross-modal enhanced CNN and Transformer module (CECT) is designed to deeply enhance the fused interactive features from global/local perspectives. Experiments show that the proposed method is superior and outperforms the comparison methods by an average of 3.06% in OA on Houston2013 and 1.79% on Summer, respectively.

Underwater images often reveal color distortion and poor visibility due to light propagation in water being affected by the selective absorption and scattering of suspended particles. This letter presents an efficient color channel transfer (CCT) method that largely restores color distortion and improves visibility of underwater images. Any captured underwater image with at least one color channel is highly attenuated in real underwater imaging. To compensate for the loss of information in the attenuated channel, the CCT transfers the degraded image to the CIELab color space and compensates for the loss of information in the degraded image by adjusting luminance and chrominance. The reference values of the color transfer image are statistically calculated from many high-quality images to ensure a relatively balanced color distribution. Extensive experiments on three underwater image datasets show that after applying our CCT, the enhancement method leads to satisfactory results in both metric scores and runtimes.

We observe that a natural image tends to exhibit similar histograms for color channels in the RGB color space and consistent statistical estimates for color channels in the Lab color space. We refer to these observations as natural color consistencies. In contrast, we discover that an underwater image does not always follow the natural color consistencies. Different color channels in an underwater image tend to give rise to very different distributions, regardless of whether the channels are in the RGB color space or Lab color space. We refer to these observations as underwater color disparities. To enhance an underwater image to make it appear more natural, it is necessary to correct its underwater color disparities to align with the natural color consistencies. To this end, we develop an adaptive attenuated channel compensation method based on optimal channel precorrection and a salient absorption map-guided fusion method for eliminating the color deviation in the RGB color space. We then develop a method to enhance the contrast of channel L and an adaptive color distribution specification method for improving the contrast and matching the color distribution in the Lab color space. Additionally, we develop an edge-enhanced mask fusion method for correcting blurry details. Our method is not a deep learning method but can effectively be applied to a single underwater image. The qualitative and quantitative empirical results validate that our method outperforms state-of-the-art underwater image enhancement methods. We release the reproducible code athttps://gitee.com/wanghaoupc/Underwater_Color_Disparitiesfor public evaluation.

Effective oil spill segmentation in synthetic aperture radar (SAR) images is critical for marine oil pollution cleanup, and proper image representation contributes to effective learning for accurate oil spill segmentation. In this article, we propose an effective oil spill segmentation network named SRCNet, which is constructed by leveraging seminal SAR image representation to empower the learning capability of the proposed segmentation network for accurate oil spill segmentation. Specifically, the image representation utilized in our proposed SRCNet originates from SAR imagery, modeling with the internal characteristics of oil spill SAR image data, which therefore promotes effective learning for accurate oil spill segmentation in the training process. Besides, to conduct enhanced oil spill segmentation, we construct the proposed SRCNet with a pair of deep neural nets that work in a competition manner, where one neural net strives to produce accurate oil spill segmentation maps by drawing samples from the collaborated seminal image representation, while the other tries its best to distinguish between the produced and the true segmentations. It is the competition and the image representation collaborated that drives the proposed SRCNet to operate accurate oil spill segmentation efficiently with small amount of training data. This establishes an economical and efficient way for oil spill segmentation. Additionally, to further improve the segmentation performance of the proposed SRCNet, a regularization term that penalizes the segmentation loss is devised, which encourages the produced segmentation to approach the ground-truth segmentation, promoting the segmentation capability of the proposed SRCNet for accurate oil spill segmentation. Experimental evaluations from different metrics validate the effectiveness of the proposed SRCNet for oil spill segmentation.

Precision agriculture management relies on the delineation of crop field edges. Multi-polarization SAR technology has the ability to penetrate clouds and capture morphological structures or moistures, suited for extracting crop field edges. Due to the time-dependent characteristics and phenological evolutions of crops, the methods with single-date data are difficult to detect complete edges. Moreover, the existing methods fail to extract the dynamic time-varying patterns, limiting the improvement of edge detection accuracy. Based on this, this paper proposes a novel crop field edge detection method based on the time-varying polarimetric characteristics. First, a spatial-temporal homogeneity measure is proposed to pre-identify the edge and homogenous area, for guiding the adaptive calculation of edge strength. Based on the time-series polarimetric stationarity and the trace moment estimation theory, the proposed measure enlarges the separating degree of various crop parcels. Second, a joint edge strength is proposed to enlarge strength contrast between edge and homogenous area. With the spatial-temporal homogeneity measure, it combines the similarity with the root mean square and the similarity with time-series average covariance matrix. Based on the advantages of two kinds of similarities, it highlights the field edges and reduces the impact of speckle noises. Evaluated by 8 quad-polarization and 14 dual-polarization SAR images, the proposed edge detection method achieves better visual presentations and detection accuracies than traditional methods. With the statistics of the signal-noise ratio (SNR), the joint edge strength also has higher strength contrast than conventional strengths. The relevant codes can be found in https://github.com/DawnHanGeo/TSPolEdge.git.

Cloud removal can effectively address cloud contamination in optical remote sensing images. But the simultaneous removal of both thin and thick clouds remains a significant challenge due to their distinct features. Thin cloud regions permit limited observation of ground information, in contrast to thick cloud, which completely obscures such information, necessitating different treatment processes. While some previous methods could effectively handle either thin or thick clouds, they often resulted in a slightly blurry outcome for the other type. In this article, we propose a cloud removal scheme that handles thin and thick clouds as a cohesive entity. Specifically, we initially use a network based on residual architecture to recover optical information under thin cloud and generate a grayscale cloud mask. Through the grayscale mask and a predefined threshold, thick cloud regions are identified. Then, an encoder-decoder network is used to estimate the information of thick cloud regions labeled by the predicted mask. Next, synthetic aperture radar (SAR) images are used as auxiliary information for cloud removal, providing the most indicative features for the edges of cloud regions. Finally, a contextual feature transfer mechanism (CFTM) imports features from remote spatial locations to fill in thick cloud regions, enhancing both visual and semantic coherence. As a result, our approach resolves the spectral defects of fuzziness and incomplete cloud removal. Experiments conducted on the SEN12MS-CR dataset confirm that our method outperforms others in all metrics, including mean absolute error (MAE), spectral angle mapper (SAM), peak signal-to-noise ratio (PSNR), and structural similarity index (SSIM).

Oil remains the most important energy source in the world today, and tankers are its main modes of transportation. However, there is a high risk of oil spills, which can cause serious damage to the ecological environment. Remote sensing monitoring is one of the most common processes of emergency management for oil spills. Short interval and uninterrupted remote sensing sequence data are crucial for monitoring and tracing oil spill events. However, the long revisit period of satellites poses a challenge of data scarcity for oil spill monitoring. To address the issue of insufficient remote sensing image data in oil spill monitoring and prediction, we propose a joint modeling approach that combines deep learning and numerical models to enhance the monitoring efficiency. First, a DiffusionLSTM network is proposed based on the diffusion model’s ability to generate images and the capability of long short-term memory (LSTM) network to extract temporal information. The proposed network can learn the evolution pattern of images from historical remote sensing data and predict future scenarios. Comparative experiments on the MODSD dataset show that our proposed model achieved a significant improvement compared to traditional time-series image prediction models (ConvLSTM and GAN-LSTM). Second, a trajectory model based on numerical simulation methods is established using OpenOil. Taking into account the differences in different marine areas, we accurately reconstruct oil spill trajectories by calibrating the wind drift coefficients. For Sanchi oil spill incident, the error is reduced approximately to 2500 m. Finally, through fusing the images generated by DiffusionLSTM and the oil spill trajectories predicted by OpenOil, short time interval oil spill scene images have been generated efficiently to improve the monitoring efficiency.

Convolutional neural network (CNN)-based hyperspectral image (HSI) classification models often exhibit high volume and complexity. This not only poses challenges in deploying them on mobile and embedded devices due to storage and power constraints but also introduces a dilemma between the growing demand for labeled samples and the high cost associated with manual labeling. To address these challenges, we propose an ultra-lightweight hybrid CNN based on redundancy removal (ULite-R2HCN), specifically designed for HSI classification in scenarios with limited samples. To reduce computational costs and enhance feature extraction effectiveness, we focus on optimizing the widely used depthwise convolution (DW-Conv) and pointwise convolution (PW-Conv) in the lightweight HSI classification model. For DW-Conv, we design a spatial convolution with redundancy removal (R2Spatial-Conv). This involves the design of multi-scale 3D convolution kernels with specific structures instead of 2D convolution kernels, aiming to reduce redundant convolution kernels and extract multi-scale spatial features. Simultaneously, for PW-Conv, we design a spectral convolution with redundancy removal (R2Spectral-Conv). This utilizes a “copy-splicing-grouping” structure to extract spectral features within arbitrary range intervals, effectively reducing redundant spectral extractions and capturing long-range spectral relationships. Numerous experiments have shown that the proposed ULite-R2HCN achieves higher classification accuracy with an ultra-light volume for a few training samples. In addition, sufficient ablation experiments also verified the advanced performance of the designed R2Spatial-Conv and R2Spectral-Conv.

Hyperspectral unmixing is a vibrant research field that focuses on the task of decomposing mixed pixels into a collection of pure spectral signatures, known as endmembers, along with their corresponding fractional abundances. Conventional unmixing algorithms often need to combine two techniques, namely endmember extraction and abundance estimation, to accomplish the unmixing task. Recently, deep learning (DL) has succeeded in the field of hyperspectral unmixing due to its strong feature learning and data-fitting capabilities. By extracting the output and weight of a particular layer as abundance maps and endmember signatures, available DL methods can directly unmix hyperspectral images. However, in order to improve the performance of spectral unmixing, such available DL methods frequently employ the results of endmember extraction algorithms –in most cases, the well-known vertex component analysis (VCA)– as the initial weights, which leads to significant limitations in their performance: a) the unmixing results are heavily dependent on the initialization given by VCA, and b) the randomness of VCA is passed to the unmixing network. In this paper, we design a new method called abundance-guided spectral and spatial network (A2SN) which not only skips the weights to extract endmember features directly from the network, but also estimates the abundance maps and reconstructs images directly. In particular, the proposed A2SN employs different kernels to capture spectral and spatial information. We also propose an abundance-guided attention spectral and spatial attention network (A2SAN) for hyperspectral unmixing by integrating attention mechanisms into A2SN. As a result, A2SAN is a completely innovative unmixing method that employs attention and reconstruction directly for hyperspectral unmixing, rather than just as modules for information extraction. Most importantly, both A2SN and A2SAN use a weighted summation of the feature maps to reconstruct the image and increase the noise immunity of the network. Experimental results, conducted on both synthetic and real datasets, demonstrate the effectiveness and superiority of A2SN and A2SAN over state-of-the-art unmixing methods. Our full code is released at https://github.com/xuanwentao/A2SN-and-A2SAN for public evaluation.

Underwater images normally suffer from visual degradation issues such as color deviations, low contrasts, and blurred details. Recently, numerous underwater image enhancement algorithms have been proposed to address these issues. However, constrained by underwater conditions, acquiring non-underwater images and depth maps for underwater images is often challenging. This limitation significantly hampers the performance of data driven-based methods and physical model-based methods. Additionally, existing physical model-based methods typically require manual parameter settings, which tend to be bruteforce and insufficient to effectively address the diverse underwater scenes. To overcome these limitations, this paper presents a comprehensive underwater image enhancement framework comprising three phases: metamergence (i.e., meta submergence), metalief (i.e., meta relief), and metaebb (i.e., meta ebb). These phases are dedicated to virtual underwater image synthesis, underwater image depth map estimation, and the configuration of state-of-the-art physical models for underwater image enhancement by reinforcement learning, separately. While the three phases are trained separately, the former phase provides the necessary data for training the latter. We refer to the overall three phases as metalantis (i.e., meta Atlantis) because its training processes, involving variations from submergence via relief to ebb over indoor scenes, mimic the virtual variations of Atlantis. The metalantis framework empowers state-of-the-art physical models of underwater imaging through reinforcement learning with virtually generated data. The well-trained metalantis framework can take an underwater image as the sole input, process it into virtual representations, and finally enhance it. Comprehensive qualitative and quantitative empirical evaluations validate that our metalantis framework outperforms state-of-the-art underwater image enhancement methods. We release our code at https://gitee.com/wanghaoupc/Metalantis_UIE.

Few-shot class-incremental learning has recently received significant research focus in remote sensing scene classification (FSCIL-RSSC). The success of FSCIL-RSSC relies on the robustness of the feature backbone and classifiers. Existing works focus on improving classifier adaptation, but little attention is paid to the importance of backbone robustness on the recognition ability of new class samples’ embeddings. Due to the large distribution shift between old and new classes, FSCIL-RSSC using high-layer (single-scale) features may not adapt flawlessly to new categories. To solve the issue, we put forward a gradient guided multiscale feature collaboration network (G-MFCN) for FSCIL-RSSC. Specifically, we introduce a parallel hierarchy strategy to simultaneously capture the multifeature discriminative information of the same sample. Then, a gradient guide block is designed to automatically pick out the optimal values of different convolution blocks for multifeature fusion. Finally, the classical feature pyramid network is introduced for multiscale fusion to obtain more obvious discriminative features of RSSC. More importantly, our proposed G-MFCN is a simple and adaptable module, which can combine any existing FSCIL frameworks to further improve the optimized classifiers’ effectiveness for the FSCIL-RSSC scenario. Extensive experiments on four benchmarks demonstrate that the proposed G-MFCN achieves significant improvements in comparison to existing FSCIL-RSSC methods.

Hyperspectral image super-resolution has been extensively studied, and significant development has been made based on deep convolutional neural networks (CNNs). Particularly, residual networks that fuse features from multiple layers have achieved high-accuracy hyperspectral image super-resolution. However, most residual networks tend to straightforwardly add features from one layer to another through skip-connections that may cause confusion about feature fusion. To tackle this issue, we develop a ConvLSTM skip-connection strategy that characterizes features from consecutive layers by ConvLSTMs and renders feature fusion in a more principal manner. Accordingly, we develop a super-resolution framework that consists of three modules. The first module, i.e., spatial feature reconstruction, employs the ConvLSTM skip-connections to comprehensively fuse spatial features from different layers. The second module, i.e., edge refinement, involves the ConvLSTM skip-connections to enhance the edge information from intermediate results. The third module, i.e., spectral information reconstruction, refines spectral features by capturing interactions between different spectral bands through the ConvLSTM skip-connections. The three complementary modules cooperate such that both spatial resolution and spectral fidelity are well maintained. Extensive experimental results on the Chikusei, Houston, and QUST-1 datasets demonstrate that our framework outperforms state-of-the-art methods in terms of quantitative evaluation and visual quality across a variety of scenarios. We release our source code athttps://gitee.com/xu_yinghao/CLSCNetfor public evaluations.

Class-incremental (CI) learning has recently received extensive research interest in remote sensing scene classification (CI-RSSC). The existing CI-RSSC methods’ superior performance seriously relies on old (base classes) and new classes (incremental classes) sampled independently from an identical distribution (dataset). In real-world RSSC scenarios, there exist significant distribution shifts between old and new classes, leading to the existing CI-RSSC methods being unable to adjust flawlessly to these new classes. In this article, we propose a novel cross-domain (CD) CI-RSSC framework to solve the above-mentioned problems, termed CDCI-RSSC. Specifically, a modular sharing-based dynamic extension module is first designed, which only updates specialized modules to extract new class feature embeddings for reducing memory footprint. Then, an effective dynamic alignment guided domain adaptive module (DAM) is further proposed to calculate the dynamic weights of each sample in various fields, which can minimize distribution shifts between source and target domains. Finally, a foreground enhancement module (FEM) is introduced to alleviate the issue of complex background interference in RSSC by increasing the weight of critical regions. Compared with the existing CI-RSSC and CD-RSSC, our proposed CDCI-RSSC framework surmounts the challenge of handling the distribution shifts between source (base session) and target domains (incremental session) while alleviating the limitations of continuous learning of new classes. Extensive experiments on three CDCI scenarios show that the CDCI-RSSC model achieves significant performance improvements in comparison to existing CI-RSSC and CD-RSSC methods.

Oceanic internal waves (IWs) are an important oceanic phenomenon, and the realization of the fast and efficient extraction of IWs is of fundamental significance. The development of deep learning techniques provides new opportunities for the signature extraction of oceanic IWs. In this letter, we propose a two-stage oceanic IW signature segmentation algorithm for synthetic aperture radar (SAR) images. The algorithm includes a decision fusion-based oceanic IW images classification stage and a pixel attention U-Net (PAU-Net)-based stripe segmentation stage. First, we adopt an IW classification algorithm by fusing the weak decision results of two different classifiers to get the final strong decision result to screen the image blocks containing oceanic IWs. Then we develop a PAU-Net to segment the IWs stripe. Finally, we concatenate them to obtain the extract results of the whole image. Experiments are performed using 527 image scenes from the Sulu Sea that contain IWs. The results show that the proposed algorithm can achieve the performance of oceanic IW signature extraction from SAR images.

Successful implementation of oil spill segmentation in synthetic aperture radar (SAR) images is vital for marine environmental protection. In this article, we develop an effective segmentation framework named DGNet, which performs oil spill segmentation by incorporating the intrinsic distribution of backscatter values in SAR images. Specifically, our proposed segmentation network is constructed with two deep neural modules running in an interactive manner, where one is the inference module to achieve latent feature variable inference from SAR images and the other is the generative module to produce oil spill segmentation maps by drawing the latent feature variables as inputs. Thus, to yield accurate segmentation, we take into account the intrinsic distribution of backscatter values in SAR images and embed it in our segmentation model. The intrinsic distribution originates from SAR imagery, describing the physical characteristics of oil spills. In the training process, the formulated intrinsic distribution guides efficient learning of optimal latent feature variable inference for oil spill segmentation. The efficient learning enables the training of our proposed DGNet with a small amount of image data. This is economically beneficial to oil spill segmentation where the availability of oil spill SAR image data is limited in practice. Additionally, benefiting from optimal latent feature variable inference, our proposed DGNet performs accurate oil spill segmentation. We evaluate the segmentation performance of our proposed DGNet with different metrics, and experimental evaluations demonstrate its effective segmentations.

Hashing for localization (HfL) is an effective method for fast localizing specific scenes in a large-scale remote sensing image. Key to its efficiency arises from a comprehensive deep hashing network that generates representational binary hash codes for image patches cropped from the remote sensing image. On the other hand, this paper will investigate the problem of encrypting the remote sensing image against the HfL task. We refer to the new task as encrypting hashing against localization (EHaL). We characterize the EHaL task in term of two cues: (I) An encrypted image patch is supposed to appear as visually similar to its original image patch as possible; (II) The hash code generated by the deep hashing network for the encrypted image patch is supposed to be not close to its original class but close to a different class. Following the two cues, we develop an encrypted patch generator, which is trained in an adversarial fashion. Based on the encrypted patch generator, we propose two remote sensing image encryption frameworks that can cause non-localization and mis-localization to the HfL task separately. Experiments validates the effectiveness of our method. Reproducible executions are given at https://github.com/JingpengHan/EHaL.

Machine-learning-based methods provide attractive solutions to algal bloom detection. However, the effective utilization of training sets remains a crucial challenge. Taking the extraction of Ulva prolifera as an example, to improve the detection accuracy, this manuscript presents a model based on self-sampling and semicorrelated co-training. The self-sampling module comprises balanced sampling and gradient descent to enhance the efficiency of extracting useful information from U.prolifera training sets. Balanced sampling optimizes the distribution of sampling points, while gradient descent determines the optimal number of sampling points. During the iteration process, useful information will be continuously extracted driven by the self-sampling module as the input of training for the subsequent machine-learning algorithm. The classical semisupervised machine-learning approach named co-training is a very effective semisupervised approach, but it requires two views to be sufficient and independent, a condition that is difficult to meet in practical applications. To address this issue, we developed a semicorrelated co-training module to achieve the two-view condition. To mitigate the problem of limited labeled samples, both labeled and unlabeled samples are used as inputs for the semicorrelated co-training module. Benefiting from the self-sampling module and the semicorrelated co-training module, the experimental results based on different U. prolifera datasets from MODIS and Sentinel-1 synthetic aperture radar (SAR) show that the proposed model in the manuscript has contributed to the improvement of the detection accuracy of U.prolifera.

In this paper, we undertake the task of fast geo-localization of a query ground image by using geo-tagged aerial images. To this end, we propose a hashing strategy that fast searches the database of geo-tagged aerial images for the ground image’s matches, whose geo-tags are exploited to estimate the ground geographic location. Specifically, we commence by converting the aerial images into ground-view aerial images that have the common angle of view (i.e., horizontal view) with the ground image. We then develop a feature extraction model and a hash encoder for generating hash codes for the images. Based on these models, the ground image and the geo-tagged aerial images are transformed to hash codes that comprehensively reflect their visual content similarity. Fast searching the geo-tagged aerial image database for the ground image’s matches is conducted subject to small Hamming distance between the hash codes. We extract a geographical cluster from the matched aerial images subject to their geo-tags. In this way, the geographic location of the ground image is efficiently retrieved according to the geographical cluster. Experiments on two datasets validate the efficiency and effectiveness of our proposed framework. We have released our implementation code at https://github.com/taoyiminR/Hashing_for_geo-localization for public evaluation.

In this paper, sea ice classification on a remote sensing image given just a small number of labeled pixels is investigated. Effective sea ice classification is rendered from two aspects. First, a feature extraction method is developed. It extracts the context feature from a classification map. Second, an iterative learning paradigm is established. The labeled pixels are divided into two training subsets. At each iteration, the context feature for one subset is extracted from the classification map which is obtained subject to the other subset. Therefore, the two subsets mutually guide each other for updating the context feature in an iterative manner, which finally renders effective sea ice classification. The above paradigm is referred to as mutually guided contexts. The advantages of the new paradigm are two-fold. First, the context feature enriches the sea ice image representation in a general manner regardless of the types of raw image data. Second, the two training subsets keep providing different refined classification maps for each other such that the comprehensiveness of the context feature is recursively enhanced. Therefore, the paradigm of mutually guided contexts comprehensively characterizes the sea ice image representation for training and classification even when only small training data are available. Experiments validate the effectiveness of the mutually guided contexts for sea ice classification.

Learning based multimodal data has attracted increasing interest in the remote sensing community owing to its robust performance. Although it is preferable to collect multiple modalities for training, not all of them are available in practical scenarios due to the restriction of imaging conditions. Therefore, how to assist the model inference with missing modalities is significant for multimodal remote sensing image processing. In this work, we propose a general framework called modality-shared hallucination network (MSH-Net) to address this issue by reconstructing complete modality-shared features from the incomplete inference modalities. Compared to conventional privilege modality hallucination methods, MSH-Net does not only help preserve the cross-modal interactions for model inference, but also scales well with the increasing number of missing modalities. We further develop a novel joint adaptation distillation (JAD) method that guides the hallucination model to learn the modality-shared knowledge from the multimodal model by matching the joint probability distributions between representation and groundtruth. This overcomes the representation heterogeneity caused by the discrepancy between inputs and structures of multimodal and hallucination model, while preserving the decision boundaries refined by multimodal cues. Finally, extensive experiments conducted on four common modality combinations demonstrate that the proposed MSH-Net can effectively address the problem of missing modalities and achieve state-of-the-art performance. Code is available at: https://github.com/shicaiwei123/MSHNet.

We explore the problem of efficiently mutually localizing panchromatic and multispectral images. We pose the problem as that of cross modal remote sensing image retrieval between panchromatic and multispectral images, and explore the employment of hash code co-construction strategy to achieve efficient retrieval. We design two special discriminative feature extractors for panchromatic and multispectral images according to the characteristics of them, and co-construct two discriminative hash code books for them. The two discriminative hash code books generate hash codes for panchromatic and multispectral images separately. Sorting the Hamming distance between the panchromatic and the multispectral image hash codes achieves efficient cross modal remote sensing image retrieval between panchromatic and multispectral images. Extensive experiments on the public data set validate the effectiveness of our method.

Hyperspectral unmixing aims at extracting pure spectral signatures and estimating their corresponding abundances at each pixel. Traditional unmixing algorithms consider end-member extraction and abundance estimation as two separate steps, and the completion of abundance estimation requires results from other endmember extraction algorithms. Considering that convolutional neural networks (CNNs) have powerful learning and data fitting capabilities, some techniques based on deep learning (DL) have been proposed in the literature. Most of them only utilize spectral information and neglect spatial information. In addition, existing unmixing methods based on DL usually extract the weight and output of a specific activation layer as endmembers and abundances, respectively. In our work, we exploit 3D convolution to propose a new 3D convolution unmixing network (3DCUN) for hyperspectral unmixing. Two types of real data, i.e., Samson and Jasper, are used to evaluate the performance of our proposed 3DCUN in endmember extraction and abundance estimation. The experimental results reflect that our proposed 3DCUN gets accurate results in estimating endmembers and abundances.

Cloud detection is a basic and important task in many high level applications of remote sensing technology. Accurate cloud detection is a challenging task. On the one hand, clouds are normally exhibited at different sizes and thicknesses. On the other hand, the boundary between the clouds and their background is usually not sharp. To address the two challenges, we present a deep learning based strategy, i.e., Gradual Boundary Net, which generates a cloud mask for detecting clouds in one cloudy image. The Gradual Boundary Net consists of two stages: (a) coarse location stage and (b) gradual boundary refinement stage. At the coarse location stage, the feature extraction network with four encoders and a cascade partial decoder (CPD) is implemented to obtain the coarse score map for locating the clouds with different sizes and thicknesses roughly. At the gradual boundary refinement stage, the coarse score map is gradually refined by a erasing and fusion strategy with several gradual boundary attention modules (GBAMs). The refined cloud mask is obtained after the two stages. The experimental results validate that our Gradual Boundary Net performs well and achieves outstanding results. The code for implementing the proposed Gradual Boundary Net is available at https://github.com/kang-wu/Gradual-Boundary-Net.

We propose the Cloudy Image Arithmetic + (CIA +) for training dataset construction of thin cloud removal, which addresses the deficiency of Cloud Image Arithmetic (CIA) that cloud shadows cannot be simulated. CIA + is able to synthesize cloudy images with cloud shadows, and as in nature, the angle and intensity of the cloud shadows vary depending on the irradiation angle and cloud thickness, which achieves state-of-the-art cloudy image synthesis. The first thin cloud removal dataset on GF satellite (TCR-GF) constructed with CIA + is released to supplement public data for cloud removal. Meanwhile, we propose the Dual-attention MSGAN to remove thin clouds. The network is capable to focus on thin cloud covered regions for the coordinate attention module encodes both channel relationship and long-range dependencies with precise position information. Several qualitative and quantitative experiments validate that the Dual-attention performs excellently on our TCR-GF dataset.

Some trajectory models on the ocean, such as oil spill dispersion forecasting models lack effective boundary constraints (Geo-fencing). Some other models use soft constraints, resulting in predicted objects that should be on the ocean often appear on land. Boundary constraints often require a lot of computational overhead because of the massive coastline spatial data. How to manage these coastline spatial data has become an urgent problem to be solved. In this paper, we propose a vector data management method based on Quadtree and vector graphics clipping algorithm to manage coastline spatial data. This method provides a solution to establish boundary constraints for trajectory model prediction by efficiently retrieving spatial data.

Endmember estimation consists of two tasks, that is, determining the number of pure spectral constituents (endmembers) and extracting their spectral signatures. We present a new geometric distance-based method for endmember estimation from hyperspectral images (HSIs), which does not need to know the number of endmembers in advance. Our strategy optimizes the widely used maximum distance analysis (MDA) method from two viewpoints. First, the traditional MDA method performs endmember estimation by computing the maximum distances between any pixel and one specific pixel, line, plane, or affine hull (AH) composed by the endmembers that have been formerly extracted. Instead, our new strategy only requires computing the maximum distance between any pixel and one specific AH. This operation provides a simpler way than MDA to estimate endmembers. Second, our strategy exploits a new distance computation between any pixel and an AH and just needs the normal vector (compared to the traditional MDA method, which uses the normal vector and offset). The new distance computation in our method is much more efficient than that in the traditional MDA method.

This work presents a novel framework, named MesoGRU, to accurately predict trajectories of mesoscale eddies (MEs), which is beneficial to study the natural ocean phenomena and perform statistical analysis of oceanic data. MesoGRU first extracts ME trajectory data and establishes two datasets according to the information of location and date in the South China Sea (SCS). Then it effectively processes SCS trajectory data and integrates SLA data and AVISO data into our combined ME dataset (CDME). Furthermore, a designated neural network with a new loss function [(weighted mean square estimation (WMSE)] is designed to learn the characteristics of ME trajectories. After thoroughly analyzing correlations of every two features, the MesoGRU network iteratively extracts key ME features and renormalizes the future time sequence of eddy trajectories. Our verification results demonstrate that MesoGRU has conducted a significant prediction improvement. The mean daily center error of ME trajectory prediction with our scheme is about 8 km and its center error for seven-day forecasting is approximately 18.2 km. Moreover, MesoGRU achieves a higher matching ratio of ME trajectory forecasting compared to other deep learning methods with a single dataset, indicating MesoGRU can predict trajectories as a state-of-the-art method.

We develop a memory graph convolutional network (MGCN) framework for sea surface temperature (SST) prediction. The MGCN consists of two memory layers: one graph layer and one output layer. The memory layer captures SST temporal changes via temporal convolution units and gate linear units. The graph layer encodes SST spatial changes in terms of characteristics derived from graph Laplacian. The output layer encapsulates information from the previous layers and produces SST prediction results. The MGCN characterizes both the temporal and spatial changes, rendering a comprehensive SST prediction strategy. We use daily mean SST data for two areas near the Bohai Sea and the East China Sea for experimental evaluations and validate that the MGCN performs better than other traditional machine learning methods for nearshore SST prediction. In addition, we test the MGCN on weekly and monthly mean SST datasets and validate that the MGCN is robust and suitable for SST prediction.

This paper presents a comprehensive underwater visual reconstruction paradigm that comprises three procedures, i.e., the E-procedure, the R-procedure, and the H-procedure. The E-procedure enhances original underwater images based on color compensation balance and weighted image fusion, yielding restored color, sharpened edges, and global contrast. The R-procedure registers multiple enhanced underwater images by exploiting global similarity and local deformation. The H-procedure homogenizes the registered underwater images by multi-scale composition strategy, which eliminates the inhomogeneous transition and brightness difference across overlapping regions, resulting in a reconstructed wide-field underwater image with comfortable and natural visibility. The three procedures operate in a cascade where the former procedure processes underwater images in a way that facilitates the latter one. We refer to the overall three procedures as the Enhancement-Registration-Homogenization (ERH) paradigm. Comprehensive qualitative and quantitative empirical evaluations reveal that our ERH paradigm outperforms state-of-the-art visual reconstruction methods, including the AutoStitch, APAP, SPHP, APNAP, and REW.

We investigate the problem of efficiently localizing sketch depicted scenes in a remote sensing image dataset. We pose the problem as that of remote sensing image retrieval with sketch queries and explore the use of hashing techniques to achieve efficient retrieval. Given two training datasets of sketches and remote sensing images that have a common set of class labels, we develop a hashing strategy that coconstructs two hash code books for the sketches and the remote sensing images separately. The hash code book coconstruction strategy encourages hash codes for the sketches and remote sensing images from different classes to be far away from one another and those from the same class to be close. This property is maintained by two cohesion intensive cues: 1) an interclass pairwise disperse cue (InterPDC) and 2) an intraclass pairwise balance cue (IntraPBC). We use the two coconstructed hash code books for training two linear mapping models that generate hash codes for sketches and remote sensing images separately. Sorting the Hamming distance between the sketch hash codes and the remote sensing image hash codes renders efficient remote sensing image retrieval with sketch queries. This enables localizing the sketch depicted scenes in the remote sensing image dataset. In addition, our method can also be used for fast localizing sketch depicted scenes in a remote sensing image of large size. Extensive experiments on public datasets validate the effectiveness and efficiency of our method.

Advanced remote-sensing instruments produce massively large scenes from the surface of the earth, with very high spatial resolution and dimensionality. Developing methods for efficiently localizing specific objects in a large-scale scene presents a significant challenge, mainly because of the high computational requirements involved. To tackle this issue, we propose a new hashing for localization (HfL) framework that efficiently searches for specific objects in the large-scale scene. It begins by dividing the scene into a large number of overlapping local patches. A lightweight deep hash model, referred to as a tiny hashing network (THNet), encodes the local patches into hash codes. The Hamming distances between the hash code of an object image, i.e., an image containing the specific class of objects to be localized in the scene, and those of all local patches are computed. Small values of the Hamming distance indicate local patches that are similar to the object image. The positions of these local patches in the large-scale scene reflect the regional locations of the specific objects. The hash codes are binary and do not take up much space, and the Hamming distance carries very low-computational overheads. Further, we exploit a class center loss as the THNet training objective, which can comprehensively manage multiple object classes. These features mean that the HfL framework can localize specific objects very quickly, regardless of the size of the scene. Extensive experiments validate the effectiveness and efficiency of the framework. For instance, HfL can find objects in a remote-sensing image of 19584$\times$19584 pixels in only 4.388 s (on a single RTX2080ti), with remarkable localization results. The source codes and datasets are available athttps://github.com/lrhan/HfL, together providing a baseline for fast localizing objects in a large-scale scene.

This article develops a general yet effective hyperspectral image (HSI) classification framework that is trained with small data. To this end, two identically structured but differently initialized classifiers, which are referred to as two base classifiers, are trained in an iterative manner. Each iteration consists of three steps, that is: 1) the two base classifiers that are trained separately on guide data; 2) unclassified data that are processed by the two trained base classifiers; and 3) the classification results with high confidence that are explored as new guide data. In the first iteration, the guide data comprising the original small training data are the same for the two base classifiers. From the second iteration, the guide data for the two base classifiers start becoming different. Specifically, in each iteration, the guide data for training one base classifier keep being augmented by high confidence classification results provided by the other base classifier. It is in such an iterative manner that the two classifiers continuously provide different new guide data for each other, and thus increasingly augment labeled data from the original small training set to a reasonably larger amount of samples in a HSI. We refer to such a training strategy as mutual guide. We develop a mutual guide implementation scheme by exploiting extreme learning machines (ELMs) as base classifiers. Extensive experiments on four public HSI datasets, i.e., Indian Pines (IP), Kennedy Space Center (KSC), University of Pavia (UP), and Salinas (SA), validate the classification effectiveness of our mutual guide framework with small training data.

Spectral unmixing plays a vital role in hyperspectral image analysis. It mainly consists of two procedures, i.e., endmember extraction and abundance estimation. Although most algorithms for each of the two procedures may exhibit good performance, few studies have been done considering both problems simultaneously. Therefore, hyperspectral unmixing accuracy is normally achieved by exploring all possible combinations of the two types of algorithms, which renders high computational overloads. We propose a novel orthogonal projection framework to conduct fast hyperspectral unmixing. It addresses both endmember extraction and abundance estimation with orthogonal projection endmember (OPE) and orthogonal projection abundance (OPA). Especially, the pixel with the largest orthogonal projection on any pixel is considered to be an endmember. We randomly choose one pixel from the hyperspectral data to compute the orthogonal projections of all pixels and extract the pixel with the largest projection as the first endmember. To avoid extracting the same endmembers, we compute orthogonal projections of all pixels to endmembers that have been previously extracted, and the pixel with the largest projection is considered as the next endmember. In terms of abundance estimation, we also utilize the concept of orthogonal projection and search for a diagonal matrix whose multiplication with the endmember matrix is not only a square matrix but also a diagonal matrix. Then, we exploit some specific matrix operations to estimate the abundance of each endmember at every pixel. We have evaluated the proposed OPE and OPA algorithms on synthetic and real data, and the experimental results have validated their effectiveness and efficiency in hyperspectral unmixing.

We present a cloudy scene synthesis paradigm that produces cloudy images for arbitrary optical cloud-free observations. The synthesis paradigm consists of two fundamental operations, i.e., 1) cloud self-subtraction and 2) cloud addition-to-scene. Cloud self-subtraction extracts cloud ingredient images from cloudy images of weak texture regions (typically sea areas). The cloud ingredient images exhibit clouds in more realistic forms than simulated clouds. Cloud addition-to-scene incorporates the cloud ingredient images into arbitrary cloud-free land images, synthesizing cloudy scenes. It provides a means of constructing data pairs of cloud-free scene images and cloudy scene images, which are highly needed but considerably insufficient in the remote sensing literature. We refer to the overall paradigm consisting of the two fundamental operations as cloudy image arithmetic. We explore the use of the cloudy image arithmetic for the purpose of thin cloud removal. To this end, we develop a multi scale generative adversarial net (MSGAN) that removes thin clouds from cloudy scenes. We use the cloudy image arithmetic to construct a comprehensive training dataset for the MSGAN. Experimental evaluations validate that the cloudy image arithmetic synthesizes good cloudy scenes and the MSGAN with aid of the cloudy image arithmetic gives effective results in thin cloud removal.

We explore self-paced boost learning (SPBL) with superpix-elwise features for hyperspectral image classification (HIC). Firstly, we conduct feature extraction using superpixelwise principal component analysis (SuperPCA), which reduces the dimensionality of hyperspectral images considering the project discrepancy in different homogeneous regions. Secondly, we perform classification by SPBL on the extracted features, where the learning just focuses on the pixels to be classified and does not make their spatial neighbours involved. SPBL embraces the power of self-paced learning on classifying from simple to complex and that of boost learning on classifying in a robust fashion. Our method is not deep learning grounded and the training does not demand high computing resources. The experimental results on two public hyperspectral image datasets demonstrate that our method is competitive with several prominent ones.

This paper proposes a multi-scale deep residual network (MDRN) for removing clouds from remote sensing images. MDRN characterizes learning the residuals of cloud-free images and cloudy images instead of directly mapping the two parts. The learned residuals are subsequently integrated with the input cloudy images to generate the de-clouded results. The advantages of our MDRN are threefold. Firstly, we untangle the sparse details of cloudy images from the bases by a guided filter, making the learning focus on processing the textures and the cloud features in the details. Secondly, we employ multi-scale convolution units (MS-Conv), which have larger receptive fields than plain convolutions and assist in extracting representative features. Thirdly, we leverage deep residual learning to avoid performance degradation and relax learning burdens. Experimental results on the remote sensing image cloud removing dataset (RICE) validate the effectiveness of our MDRN.

Sea-ice classification plays an important role in evaluating sea-ice hazards and ensuring maritime safety. In this paper, a method of sea-ice classification based on morphological feature extraction is proposed by using CEBRS-02B multispectral CCD image. The paper uses local contain profile (LCP) to extract morphological features of the multispectral image. Then SVM- and MRF- (SVMMRF) is adopted, which including probabilistic support vector machine (SVM) for the preliminary classification of multispectral image, the postprocessing by using Markov random field (MRF) based regularization. Experimental results demonstrate the validity of the classification model framework is applied to sea-ice classification. Compared with the traditional Local Binary Pattern (LBP) and Gabor, feature extraction by LCP can improve the accuracy of sea-ice classification.

We explore the use of convolutional neural networks (CNNs) for filling voids in digital elevation models (DEM). We propose a baseline approach using a fully convolutional network to predict complete from incomplete DEMs, which is trained in a supervised fashion. We then extend this to a shadow-constrained CNN (SCCNN) by introducing additional loss functions that encourage the restored DEM to adhere to geometric constraints implied by cast shadows. At the training time, we use automatically extracted cast shadow maps and known sun directions to compute the shadow-based supervisory signal in addition to the direct DEM supervision. At the test time, our network directly predicts restored DEMs from an incomplete DEM. One key advantage of our SCCNN model is that it is characterized by both CNN data inference and geometric shadow cues. It thus avoids data restoration that may violate shadowing conditions. Both our baseline CNN and SCCNN outperform the inverse distance weighting (IDW)-based interpolation method, with the shadow supervision enabling SCCNN to obtain the best performance.

We develop a hierarchical generative adversarial network (HGAN) for generating future typhoon cloud remote sensing images, which enables a visual means to typhoon cloud prediction. The HGAN consists of a global generator and a local discriminator. The global generator aims at producing the future typhoon cloud images as realistic as possible and accordingly reveals the structure and future location of the typhoon clouds. It is constructed in terms of a hierarchical architecture with multiple subnetworks, which capture the overall typhoon variations and favor generating clear future typhoon cloud images. The local discriminator tries its best to distinguish generated typhoon cloud images from ground-truth ones, based on the local patches. The local procedure encourages the discriminator to focus on characterizing the moving typhoon clouds rather than the still background. The global generator and the local discriminator are trained in an adversarial fashion with respect to historical typhoon cloud image sequences. The trained HGAN is capable of producing reliable visual predictions that are not only enabled by the global generator and but also examined by the local discriminator. Experiments validate the effectiveness of the HGAN for typhoon cloud prediction.

Fast and accurate remote sensing image retrieval from large data archives has been an important research topic in the remote sensing research literature. Recently, hashing-based remote sensing image retrieval has attracted extreme attention because of its efficient search capabilities. Especially, deep remote sensing image hashing algorithms have been developed based on convolutional neural networks (CNNs) and have shown effective retrieval performance. However, implementing a deep hashing network tends to be highly expensive in terms of storage space and computing resources to be suitable for on-orbit remote sensing image retrieval, which usually operates on resource-limited devices such as satellites and unmanned aerial vehicles (UAVs). To address this limitation, we propose to hash a deep network that in turn hashes remote sensing images. Specifically, we develop a quantized deep learning to hash (QDLH) framework for large-scale remote sensing image retrieval. The weights and activation functions in the QDLH framework are binarized to low-bit representations, which require comparatively much less storage space and computing resources. The QDLH results in a lightweight deep neural network for effective remote sensing image hashing. We conduct extensive experiments on two public remote sensing image data sets by incorporating several state-of-the-art network architectures into our QDLH methodology for remote sensing image hashing. The experimental results demonstrate that the proposed QDLH is effective in saving hardware resources in terms of both storage and computation. Moreover, superior remote sensing image retrieval performance is also achieved by our QDLH, compared with state-of-the-art deep remote sensing image hashing methods.

Small-aperture compact high-frequency surface wave radar (HFSWR) suffers from low azimuth accuracy for target detection due to its wide beamwidth. Multitarget tracking (MTT) algorithms, when applied to the raw target detection data of HFSWR, fail to effectively filter the target azimuths, and thus, resulting in inaccurate target tracks and courses. In this article, a vessel azimuth and course joint re-estimation method by exploring Doppler velocity and the information accumulated from consecutive observations is presented. It begins with applying an MTT algorithm to a measured target states data sequence acquired by HFSWR to establish initial target tracks, from which the measured range, azimuth, and radial velocity data sequences are obtained. Then, the azimuth trend is extracted from the obtained azimuth data sequence as roughly corrected azimuth estimates, with which the target locations are roughly corrected. Subsequently, target speeds and initial courses are estimated based on the roughly corrected location data sequence, followed by a data selection procedure based on proposed control parameter rules to select the qualified data for calculating the projected angles in terms of speed and direction, separately. Eventually, the target azimuth data sequence is further refined using a linear azimuth error model, whose parameters are obtained by minimizing the difference between the projected angles using a constrained optimization method. Experimental results from field data demonstrate that the proposed method can estimate the target azimuths with significantly improved accuracy. The deviations of the corrected target locations are considerably reduced, and the accuracy of course estimation is enhanced.

Most existing endmember extraction techniques require prior knowledge about the number of endmembers in a hyperspectral image. The number of endmembers is normally estimated by a separate procedure, whose accuracy has a large influence on the endmember extraction performance. In order to bridge the two seemingly independent but, in fact, highly correlated procedures, we develop a new endmember estimation strategy that simultaneously counts and extracts endmembers. We consider a hyperspectral image as a hyperspectral pixel set and define the subset of pixels that are most different from one another as the divergent subset (DS) of the hyperspectral pixel set. The DS is characterized by the condition that any additional pixel would increase the likeness within the DS and, thus, reduce its divergent degree. We use the DS as the endmember set, with the number of endmembers being the subset cardinality. To render a practical computation scheme for identifying the DS, we reformulate it in terms of a quadratic optimization problem with a numerical solution. In addition to operating as an endmember estimation algorithm by itself, the DS method can also co-operate with existing endmember extraction techniques by transforming them into a novel and more effective schemes. Experimental results validate the effectiveness of the DS methodology in simultaneously counting and extracting endmembers not only as an individual algorithm but also as a foundation algorithm for improving existing methods. Our full code is released for public evaluation.

We explore the use of graphical generative adversarial networks (Graphical-GAN) for synthesizing remote sensing images. The model is probabilistic graphical based generative adversarial networks (GAN). It pairs a generative network G with a recognition network R. Both of them are adversarially trained with a discriminative network D. Particularly, R is employed to infer the underlying causal relationships among both observed and latent variables from real remote sensing images. The advantages of the Graphical-GAN for synthesizing multiple categories of remote sensing images are two fold. Firstly, it considers the underlying causal relationships and captures the true data distribution of remote sensing images. Secondly, the adversarial learning generates synthetic sensing images that are similar to real ones with slight differences. Our remote sensing image synthesis scheme paves a promising way for remote sensing dataset augmentation, which is an effective means of improving the accuracy of learning models. Experimental results with high Inception Scores (IS) validate the effectiveness of the Graphical-GAN for remote sensing image synthesis.

Existing scene classification models tend to be only capable of recognizing scene images from classes which have been learned by the models. This implies that testing images from classes which are not used for training the models cannot be recognized. There are tens of thousands of scene classes in the real world and it is quite infeasible for us to label all these classes for model training. Therefore, how to develop a classification model that recognizes scene images from unseen classes has been an open problem. To address this issue, we investigate the underlying relations between scene attributes and scene classes. We observe that one scene class can be characterized by a few scene attributes, and furthermore, one unseen scene class can be depicted by seen attributes. In the light of this observation, we train support vector machines to classify scene attributes rather than scene classes. We determine the attributes of a scene image from an unseen class by the support vector machines. This enables us to infer the unseen class of the scene image with respect to the attribute-class relations. Experimental results validate our framework.

We present a cofactor-based endmember extraction strategy for estimating green algae area in geostationary ocean color imager multispectral images. Our strategy improves the efficiency of the widely used N-FINDR endmember extraction method from two aspects. First, our strategy exploits the cofactor matrix for searching the largest simplex volume, which just computes matrix inverse and determinant for a small number of times (or even once). This is more efficient than the enumeration of determinants for all pixels in N-FINDR. Second, our strategy empirically obtains optimal endmembers through a few recursive iterations of cofactor matrix updates, contrasting a large number of repetitive volume maximizations with random initializations in N-FINDR. Experimental evaluation in terms of green algae area estimation validates that our strategy achieves the same accuracy as N-FINDR with much more efficiency.

Object detection plays an active role in remote sensing applications. Recently, deep convolutional neural network models have been applied to automatically extract features, generate region proposals, and predict corresponding object class. However, these models face new challenges in VHR remote sensing images due to the orientation and scale variations and the cluttered background. In this letter, we propose an end-to-end multiscale visual attention networks (MS-VANs) method. We use skip-connected encoder-decoder model to extract multiscale features from a full-size image. For feature maps in each scale, we learn a visual attention network, which is followed by a classification branch and a regression branch, so as to highlight the features from object region and suppress the cluttered background. We train the MS-VANs model by a hybrid loss function which is a weighted sum of attention loss, classification loss, and regression loss. Experiments on a combined data set consisting of Dataset for Object Detection in Aerial Images and NWPU VHR-10 show that the proposed method outperforms several state-of-the-art approaches.

Learning to hash is a popular technique for fast similarity search on a large-scale image database. However, many hashing methods do not achieve satisfactory results because of the quantization loss in the straightforward binary code generation procedure. In order to address this problem, we propose a novel hash code reconstruction framework for existing unsupervised hashing methods. In our proposed approach, the hash codes are generated through reconstructing the original images with relaxed hamming vector representation, such that the final learned codes will be more approximate to characterize the intrinsic image structure. Moreover, our proposed hash code reconstruction algorithm is very efficient for computing, which can be generalized to various hashing methods. Extensive experiments are conducted on four public image datasets by incorporating our proposed scheme with different hashing methods, and the comparison results have shown that significant performance improvements can be achieved with minor additional time cost for fast similarity search task.

We develop conditional adversarial networks (CAN) framework for filling Shuttle Radar Topography Mission (SRTM) void data. We train a CAN model in terms of using incomplete and complete terrain images as inputs and outputs, respectively. In this scenario, the CAN model characterizes a void-to-filling translation and thus learns the knowledge for void data restoration. Furthermore, in order to make the void-filled images more realistic and less blurry, we employ a L1 norm to constrain the CAN training process. The trained CAN is used for restoring the incomplete SRTM data. Experimental comparisons reveal that our framework outperforms the interpolation strategy. Additionally, experiments also validate that our method performs well in restoring large areas of SRTM missing data, such as rectangular areas from 36°N74°Eto 37°N75°E.

We present a novel typhoon cloud prediction method via generative adversarial networks (GANs). Specifically, we develop a adversarial prediction model consisting of a generator and a discriminator. The generator generates the continuous future cloud images by learning the evolution trend of typhoon clouds from multiple continuous historical typhoon cloud images. In this way, the generator completes the visual predictions for typhoon clouds. On the other hand, the discriminator distinguishes the generated future cloud images from the real ones. Furthermore, we adopt a gradient difference loss function and a total variation loss function to improve the quality of the generated cloud images. The proposed method effectively predicts the whole spatial-temporal evolution of the typhoon clouds, resulting in a visual complement for classic typhoon prediction methods. The effectiveness of the proposed method has been evaluated in the real satellite cloud images.

We develop an aerial image super-resolution method by training convolutional neural networks (CNNs) with respect to wavelet analysis. To this end, we commence by performing wavelet decomposition to aerial images for multiscale representations. We then train multiple CNNs for approximating the wavelet multiscale representations, separately. The multiple CNNs thus trained characterize aerial images in multiple directions and multiscale frequency bands, and thus enable image restoration subject to sophisticated culture variability. For inference, the trained CNNs regress wavelet multiscale representations from a low-resolution aerial image, followed by wavelet synthesis that forms a restored high-resolution aerial image. Experimental results validate the effectiveness of our method for restoring complicated aerial images.

We present a two-fold fusion framework that constructs comprehensive cost volumes for stereo matching. To this end, we develop fusion schemes at two key steps, i.e., a) the raw cost computation and b) the cost aggregation. Specifically, we commence by fusing both structure- and data-oriented features as raw costs. We then incorporate the guided filtered costs into the cross-based cost aggregation and obtain the fused aggregated costs. The fusion schemes at the two steps effectively complement each other and result in an accurate disparity map. Experiments on the Middlebury benchmark v3 demonstrate the state-of-the-art performance of our framework in terms of various metrics.

We develop an automatic oil spill segmentation method in terms of f-divergence minimization. We exploit f-divergence for measuring the disagreement between the distributions of ground-truth and generated oil spill segmentations. To render tractable optimization, we minimize the tight lower bound of the f-divergence by adversarial training a regressor and a generator, which are structured in different forms of deep neural networks separately. The generator aims at producing accurate oil spill segmentation, while the regressor characterizes discriminative distributions with respect to true and generated oil spill segmentations. It is the coplay between the generator net and the regressor net against each other that achieves a minimal of the maximum lower bound for the f-divergence. The adversarial strategy enhances the representational powers of both the generator and the regressor and avoids requesting large amounts of labeled data for training the deep network parameters. In addition, the trained generator net enables automatic oil spill detection that does not require manual initialization. Benefiting from the comprehensiveness of f-divergence for characterizing diversified distributions, our framework can accurately segment variously shaped oil spills in noisy synthetic aperture radar images. Experimental results validate the effectiveness of the proposed oil spill segmentation framework.

Segmentation of marine oil spill regions from SAR images has always been an important research topic in the remote sensing literature. One challenge for this problem is how to precisely preserve oil spill edges in segmentation results. To address this challenge, we present an edge sensitive algorithm for marine oil spill segmentation based on an energy minimization formulation. Specifically, we use an edge indicator in penalty terms of the energy function, for the purpose of detecting possible oil spill edges. Furthermore, a self-guided filtering scheme is incorporated into the energy function, which is capable of smoothing images without blurring edges. The energy minimization is conducted based on level set evolution, in which a double well distance regularization is involved for avoiding evolutionary irregularities. Empirical evaluations reveal that our method outperforms state of the art edge-based level set methods in marine oil spill segmentation.

With the rapid progress of satellite and aerial vehicle technologies, large-scale remote sensing (RS) image retrieval has recently become an important research issue in geosciences. Hashing-based searching approaches have been widely employed in content-based image retrieval tasks. However, most hash schemes compromise between learning efficiency and retrieval accuracy, and can thus barely satisfy the precise requirements in RS data analysis. To address these shortcomings, we introduce a partial randomness scheme for learning hash functions, which is referred to as partial randomness hashing (PRH). Specifically, for constructing hash functions, a part of model parameter values are randomly generated and the remaining ones are trained based on RS images. The randomness enables an efficient hash function construction and the trained model parameters encode characteristics from RS images. The coplay between random and trained model parameters results in both efficient and effective learning scheme for constructing hash functions. Experiments on two large public RS image data sets have shown that our PRH method outperforms state of the arts in terms of both learning efficiency and retrieval accuracy.

We present a novel marine oil spill segmentation method that characterizes two smoothing modules at the label level and the pixel level separately. At the label level, we exploit the rolling guidance filter for smoothing the label cost volumes. It enables scale-aware labeling and thus alleviates the ambiguous segmentation that blurs the detailed structures of oil spills. At the pixel level, we adapt a cooperative model for smoothing higher order pixel variations, which has the potential of preserving elongated strips that often arise in oil spills. We integrate the two smoothing modules operating at different levels into an energy minimization formulation, which is referred to as dual smoothing. The coupling of the two smoothing modules enables an effective complement to each other such that the specific structures of oil spills are accurately characterized. We compute the optimal labeling of the dual-smoothing framework based on graph cuts. The proposed dual-smoothing framework is especially effective in segmenting elongated and detailed oil spills, and the experimental results demonstrate its advantages over thresholding- and graph-cut-based segmentations.

We present a novel band weighting strategy that exploits multiple binary support vector machines (SVMs) to maximize interclass spectral distances for multiclass hyperspectral remote image classification. Specifically, we commence by training binary SVMs based on the original training samples. We then balance the bands of training samples by maximizing the modified classification scores for SVMs. This balance scheme enlarges the distances between individual training samples and the SVM hyperplane. For each class, we reformulate the binary SVM objective function based on the balanced training samples, resulting in a weighting vector that associates a weight to each spectral band for the class. For a testing sample, we weight it and then classify it by using the binary SVM, both with respect to every individual class. The classification result is obtained from the classifier with the greatest score. Experiments on two benchmark data sets show the effectiveness of the proposed strategy.

Various types of features can be extracted from very high resolution remote sensing images for object classification. It has been widely acknowledged that the classification performance can benefit from proper feature fusion. In this letter, we propose a softmax regression-based feature fusion method by learning distinct weights for different features. Our fusion method enables the estimation of object-to-class similarity measures and the conditional probabilities that each object belongs to different classes. Moreover, we introduce an approximate method for calculating the class-to-class similarities between different classes. Finally, the obtained fusion and similarity information are integrated into a marginalized kernel to build a support vector machine classifier. The advantages of our method are validated on QuickBird imagery.

Hashing is very useful for fast approximate similarity search on large database. In the unsupervised settings, most hashing methods aim at preserving the similarity defined by Euclidean distance. Hash codes generated by these approaches only keep their Hamming distance corresponding to the pairwise Euclidean distance, ignoring the local distribution of each data point. This objective does not hold for k-nearest neighbors search. In this paper, we firstly propose a new adaptive similarity measure which is consistent with k-NN search, and prove that it leads to a valid kernel. Then we propose a hashing scheme which uses binary codes to preserve the kernel function. Using low-rank approximation, our hashing framework is more effective than existing methods that preserve similarity over arbitrary kernel. The proposed kernel function, hashing framework, and their combination have demonstrated significant advantages compared with several state-of-the-art methods.

In information retrieval, efficient accomplishing the nearest neighbor search on large scale database is a great challenge. Hashing based indexing methods represent each data instance as a binary string to retrieve the approximate nearest neighbors. In this paper, we present a semi-randomized hashing approach to preserve the Euclidean distance by binary codes. Euclidean distance preserving is a classic research problem in hashing. Most hashing methods used purely randomized or optimized learning strategy to achieve this goal. Our method, on the other hand, combines both randomized and optimized strategies. It starts from generating multiple random vectors, and then approximates them by a single projection vector. In the quantization step, it uses the orthogonal transformation to minimize an upper bound of the deviation between real-valued vectors and binary codes. The proposed method overcomes the problem that randomized hash functions are isolated from the data distribution. What's more, our method supports an arbitrary number of hash functions, which is beneficial in building better hashing methods. The experiments show that our approach outperforms the alternative state-of-the-art methods for retrieval on the large scale dataset.

In this paper, we aim to characterize hyper graphs in terms of structural complexities. To measure the complexity of a hyper graph in a straightforward way, we transform a hyper graph into a line graph which accurately reflects the multiple relationships exhibited by the hyper graph. To locate the dominant substructure within a line graph, we identify a centroid vertex by computing the minimum variance of its shortest path lengths. A family of directed centroid expansion sub graphs of the line graph is then derived from the centroid vertex. We compute the directed depth-based complexity trace of a hyper graph by measuring directed entropies on its directed sub graphs. The novel hyper graph complexity trace provides a flexible framework that can be applied to both hyper graphs and graphs. Experiments on standard (hyper)graph datasets demonstrate the effectiveness and efficiency of the new complexity trace.

In this paper, we present a hyper graph kernel computed using substructure isomorphism tests. Measuring the isomorphisms between hyper graphs straightforwardly tends to be elusive since a hyper graph may exhibit varying relational orders. We thus transform a hyper graph into a directed line graph. This not only accurately reflects the multiple relationships exhibited by the hyper graph but is also easier to manipulate isomorphism tests. To locate the isomorphisms between hyper graphs through their directed line graphs, we propose a new directed Weisfeiler-Lehman isomorphism test for directed graphs. The new isomorphism test precisely reflects the structure of the directed edges. By identifying the isomorphic substructures of directed graphs, the hyper graph kernel for a pair of hyper graphs is computed by counting the number of pair wise isomorphic substructures from their directed line graphs. We show that our kernel limits tottering that arises in the existing walk and sub tree based (hyper)graph kernels. Experiments on challenging (hyper)graph datasets demonstrate the effectiveness of our kernel.

The p-stable distribution is traditionally used for data-independent hashing. In this paper, we describe how to perform data-dependent hashing based on p-stable distribution. We commence by formulating the Euclidean distance preserving property in terms of variance estimation. Based on this property, we develop a projection method, which maps the original data to arbitrary dimensional vectors. Each projection vector is a linear combination of multiple random vectors subject to p-stable distribution, in which the weights for the linear combination are learned based on the training data. An orthogonal matrix is then learned data-dependently for minimizing the thresholding error in quantization. Combining the projection method and orthogonal matrix, we develop an unsupervised hashing scheme, which preserves the Euclidean distance. Compared with data-independent hashing methods, our method takes the data distribution into consideration and gives more accurate hashing results with compact hash codes. Different from many data-dependent hashing methods, our method accommodates multiple hash tables and is not restricted by the number of hash functions. To extend our method to a supervised scenario, we incorporate a supervised label propagation scheme into the proposed projection method. This results in a supervised hashing scheme, which preserves semantic similarity of data. Experimental results show that our methods have outperformed several state-of-the-art hashing approaches in both effectiveness and efficiency.

The essential broadcasting feature of radio propagation channels provides an opportunity for multiple nodes to exchange information and work cooperatively, where each node, for example, mobile sensor or robot, plays both the role of transmitter and receiver. Especially in such machine‐to‐machine communication scenarios, the network performance and redundancy of information from different providers highly affect the work efficiency of each individual node and the whole system. To address these problems, a relay assisted centralised network model with physical layer network coding implemented in the relay is proposed in this study. This structure has the advantage of flexible data exchange and the capability to reduce redundancy in information. Its theoretical performance is analysed by the diversity multiplexing tradeoff, which proves the proposed model is versatile in reliable and high spectral‐efficiency information exchange. Experiments of multiple nodes in a machine‐to‐machine scenario – unmanned aerial vehicles, further reveal its potential in improving efficiency of communication and cooperation.

Most existing feature selection methods focus on ranking individual features based on a utility criterion, and select the optimal feature set in a greedy manner.However, the feature combinations found in this way do not give optimal classification performance, since they tend to neglect the correlations among features.In an attempt to overcome this problem, we develop a novel unsupervised feature selection technique by using hypergraph spectral embedding, where the projection matrix is constrained to be a selection matrix designed to select the optimal feature subset.Specifically, by using multidimensional interaction information (MII) as a higher order similarity measure, we establish a novel hypergraph framework which is used for characterizing the multiple relationships within a set of samples.Thus, the structural information latent in the data can be more effectively modeled.We then derive a hypergraph embedding view of feature selection which casts the feature discriminant analysis into a regression framework that considers the correlations among features.Within our framework, features are evaluated in combinations rather than considered individually, and feature redundancies can thus be addressed accordingly.Experimental results demonstrate the effectiveness of our feature selection method on a number of standard datasets.

The novel contributions of this paper are twofold. First, we demonstrate how to characterize unweighted graphs in a permutation-invariant manner using the polynomial coefficients from the Ihara zeta function, i.e., the Ihara coefficients. Second, we generalize the definition of the Ihara coefficients to edge-weighted graphs. For an unweighted graph, the Ihara zeta function is the reciprocal of a quasi characteristic polynomial of the adjacency matrix of the associated oriented line graph. Since the Ihara zeta function has poles that give rise to infinities, the most convenient numerically stable representation is to work with the coefficients of the quasi characteristic polynomial. Moreover, the polynomial coefficients are invariant to vertex order permutations and also convey information concerning the cycle structure of the graph. To generalize the representation to edge-weighted graphs, we make use of the reduced Bartholdi zeta function. We prove that the computation of the Ihara coefficients for unweighted graphs is a special case of our proposed method for unit edge weights. We also present a spectral analysis of the Ihara coefficients and indicate their advantages over other graph spectral methods. We apply the proposed graph characterization method to capturing graph-class structure and clustering graphs. Experimental results reveal that the Ihara coefficients are more effective than methods based on Laplacian spectra.

We develop a novel method for extracting graph characteristics from edge-weighted graphs, based on an extension of the Ihara zeta function from unweighted to edge-weighted graphs. This is effected by generalizing the determinant form of the Ihara zeta function. We use the set of the reciprocal polynomial coefficients of the resulting Ihara zeta function, i.e. the Ihara coefficients, to construct our characterization. We also present a spectral analysis of the edge-weighted graph Ihara coefficients and indicate their advantages over graph spectral methods. Experimental results reveal that the Ihara coefficients are effective for the purpose of clustering edge-weighted graphs.

This paper shows how to construct pattern vectors from the Ihara zeta function for the purposes of characterizing graph structures. To avoid the risk of sampling the meaningless infinities at the poles of the Ihara zeta function, we take use of the coefficients of the polynomial of the reciprocal zeta function. The proposed pattern vector is proved to be permutation invariant to the node order of the associated graph. Its components can be computed from a characteristic polynomial derived from the original graph. We apply the proposed scheme to graph clustering.

Local invariant feature based methods have been proven to be effective in computer vision for object recognition and learning. But for an image, the number of points detected and to be matched may be very large, or even redundantly represent the shape information present. Since selective attention is a basic mechanism of the visual system, we explore whether there is a subset of salient points that can be robustly detected and matched. We propose a method to rank the redundant local invariant features. The results prove that the top ranked points capture the salient information effectively. The method can be used as a pre-processing step for the Bag-of-Feature based methods or graph based methods. Here they simplify the complexity of the processes, such as training, matching and tracking.