The self-supervised contrastive learning strategy has attracted considerable attention due to its exceptional ability in representation learning. However, current contrastive learning tends to learn global coarse-grained representations of the image that benefit generic object recognition, whereas such coarse-grained features are insufficient for fine-grained visual recognition. In this paper, we incorporate subtle local fine-grained feature learning into global self-supervised contrastive learning through a pure self-supervised global-local fine-grained contrastive learning framework. Specifically, a novel pretext task called local discrimination (LoDisc) is proposed to explicitly supervise the self-supervised model’s focus toward local pivotal regions, which are captured by a simple but effective location-wise mask sampling strategy. We show that the LoDisc pretext task can effectively enhance fine-grained clues in important local regions and that the global-local framework further refines the fine-grained feature representations of images. Extensive experimental results on different fine-grained object recognition tasks demonstrate that the proposed method can lead to a decent improvement in different evaluation settings. The proposed method is also effective for general object recognition tasks.

Deep cross-modal hashing has gained significant attention because of its benefits, including reduced storage requirements and enhanced retrieval efficiency. Although progress has been made, existing deep cross-modal hashing methods still face unresolved challenges. Most existing methods typically adopt Euclidean space as the embedding space to measure the semantic similarity between original samples. However, the volume of Euclidean space grows polynomially with dimension, which exacerbates the curse of dimensionality. In contrast, methods based on spherical space usually use cosine similarity as the metric, effectively mitigating the aforementioned problem by normalizing the embedding vectors. Nevertheless, such methods only considers the direction to determine the category, ignoring the uncertainty measure in the embedding space, thus having a limited ability to preserve inherent multimodal semantics. In this paper, with a novel extension of the maximum entropy distribution on the surface of a hypersphere von Mises-Fisher (vMF) distribution, a novel deep cross-modal hashing method, named Deep Stochastic Spherical Hashing (DSSH), is designed to utilize uncertain information to guide the hashing process and produce discriminative modality-invariant hash codes. Specifically, to learn explicit uncertainty in learned embedding space, the Spherical von Mises-Fisher distribution is applied for the f irst time in deep cross-modal hashing, where the direction of the sample embedding controls its position on the hyper sphere, thereby preventing its semantic content, and its norm parameterizes the determinism of the distribution. In addition, stochastic spherical von Mises–Fisher loss is proposed to preserve the mode-specific semantic information of the sample, achieving the alignment of different modalities and semantic embeddings. Extensive experiments on four benchmark datasets show that our DSSH framework outperforms existing state-of-the-art cross modal hashing methods. The source code of the experiments is available at https://github.com/QinLab-WFU/DSSH.

Cloud-induced gaps in Sea Surface Temperature (SST) pose significant challenges to climate modeling and oceanographic research by causing the loss of critical spatiotemporal data. Recently, traditional deep neural network-based methods have shown promising results by effectively utilizing rich historical data. However, when confronted with extensive gaps in SST data, methods such as CNN, LSTM, and Transformer struggle to model long-range dependencies effectively or often overlook non-salient features in the gap inpainting process. To address this, we introduce a novel approach for SST completion under large-scale missing data conditions, focusing on modeling long-range dependencies by leveraging Mamba to adaptively select essential features. We propose an innovative Mamba-based inpainting network, DINMamba, for reconstructing large missing areas in SST. Specifically, to ensure physical coherence across scales, DINMamba designs a stackable recursive hierarchical Mamba block that progressively captures both stable patterns and anomalies. Finally, the fusion of these two types of information guarantees more coherent and accurate reconstructions across extensive missing regions. Experimental results on real-world datasets demonstrate the superiority of DINMamba over several state-of-the-art (SOTA) methods.Under a$68\%$missing rate, our model achieves significant improvements, reducing$RMSE$by$34.07\%$,$34.91\%$, and$37.23\%$, enhancing$R^{2}$by$5.78\%$,$7.06\%$, and$8.17\%$, increasing$SSIM$by$26.73\%$,$28.82\%$, and$31.17\%$, and boosting$PSNR$by$4.93\%$,$5.01\%$, and$5.97\%$, across noise-to-signal (N/S) ratios of 0.1, 0.2, and 0.3, respectively. Furthermore, ablation studies validate the contribution of each component and the model’s generalization capability across different seasons.

Cross-modal hashing seeks to encode heterogeneous image-text data into compact binary codes for efficient retrieval. While significant efforts have been made to study sampling strategies, most of these approaches are tightly integrated with loss function engineering, lacking an independent focus on sampling methods. In this paper, we challenge the convention by revealing that batch-level sampling strategy is equally pivotal as loss design for learning discriminative hash codes. Specifically, by introducing a novel distribution-aware sampling strategy, a Distance Weighted Sampling Hashing (DDWSH) framework is proposed to dynamically select stable and informative training pairs. Unlike conventional random or semi-hard sampling, our method weights pairwise distances within each batch to approximate global data distribution, thereby mitigating training instability caused by biased sampling. To rigorously validate our claims, we conduct the first comprehensive crossover study between sampling strategies (random/semi-hard/ours) and loss functions (contrastive/triplet/ours) across three benchmark datasets. Experiments demonstrate that: Universality: Our sampling boosts all loss functions' performance (average +4.3% mAP vs. semi-hard mining), Superiority: DDWSH competes with complex loss function design and achieves state-of-the-art results, and Stability: It reduces performance variance by 60.2% compared to semi-hard sampling under varying batch compositions average. This systematic analysis establishes sampling as an independent research dimension in deep hashing, beyond a mere part of loss function engineering. The source code for DDWSH is freely available athttps://github.com/QinLab-WFU/DDWSH.

Because of fast retrieval speed and low storage cost, deep hash learning has become a current research spot in multimedia retrieval, which arouses more and more interest and attention. Most of the available deep hashing methods usually employ the mini-batch strategy to construct training batches, and only one mini-batch of samples is valuable at each iteration during model optimization, which fails to explore the original neighbor structure well, leading to sub-optimal embeddings, especially for relatively large datasets. By contrast, the superior performance is achieved by optimizing the SoftMax loss for certain hashing learning, meanwhile, prior research has suggested the normalized SoftMax loss is essentially identical to one smoothed triplet-wise function, in which each class is assigned to one single semantic center. Nevertheless, under real-world scenarios, one class could correspond to multiple local clusters rather than just a single one. To this end, by extending SoftMax loss with multiple semantic centers, a novel deep hashing framework, called Deep Neighbor Discriminant Binary Embedding (NDBE) framework, is presented to generate discriminative hash codes with the original neighbor structure preservation. Specifically, by expanding the size of the last fully connected layer to increase multiple centers for each class, a novel SoftTriple loss is proposed to capture the latent semantic distribution of the original samples and reduce the intra-class variance, which is optimized without constraints sampling. To learn the different numbers of each class center, the class-center aggregation strategy is developed to obtain the compact set of centers while maintaining semantic similarity. Extensive experiments on three public multi-label datasets show that our proposed NDBE framework achieves superior visual similarity search performance over several state-of-the-art approaches. The source code for the implementation of our proposed NDBE framework is available athttps://github.com/QinLab-WFU/NDBE.

With low storage cost and high retrieval efficiency, hashing techniques are widely used for multi-media retrieval, which has already become the present research focus. Currently, cross-modal hashing commonly employs graph-based loss to construct pair-wise semantic relations between training samples for model optimization. However, limited by the graph-based strategy, each edge in the graph only connects two samples, which only represent a bundle of pair-wise relationships. Besides, the edges in the graph are calculated by self-attention or feature distance, only considering pair-wise relations of heterogeneous samples and ignoring the class relations. In this paper, by hypergraph modeling the semantic tuples, a novel Deep Semantic Tuplet-based Hashing by Hypergraph Modeling (DSTH) approach is proposed to leverage the multilateral semantic relations, which could guide the model to learn class-discriminative semantic binary embedding. In more detail, based on the characteristic distribution, semantic tuples are constructed for each class in one mini-batch, which represents the multilateral semantic relationships between multiple samples and multiple classes. By considering semantic tuples as hyperedges to represent multilateral semantic relations, hypergraph modeling is designed, in which HyperGraph Neural Hetwork (HGNH) is introduced to formulate hypergraph node classification goals to fully learn the multilateral semantic information contained in the semantic tuples. Moreover, to utilize the heterogeneity of local structures in embedding, the adaptive neighborhood structure is explored by learning the structure embedding, which provides fine-grained ranking lists. Through extensive experiments on three benchmark datasets, the comprehensive results validate the advancement of our proposed DSTH framework over mainstream cross-modal hashing. The source code for the framework DSTH is freely available athttps://github.com/QinLab-WFU/DSTH.

Neural operators, such as graph neural operators (GNOs) and Fourier neural operators (FNOs), directly learn the mapping from any functional parametric dependence to the solution and have achieved remarkable progress in solving partial differential equations (PDEs). GNOs exhibit excellent interpretability, as they construct graph models of physical fields to mine the mutual relationships between different nodes. However, existing methods are unable to mine the deep-level graph node features, which leads to insufficient information from neighboring nodes during the graph information aggregation process, ultimately resulting in a decline in solution accuracy. This article proposes a space-frequency cross-attention (CA) node feature optimization GNO (NFO-GNO) to address this issue. Considering the multiscale nature of the PDEs, we first construct a multiscale graph building module to obtain the PDEs information at different scales by processing graphs of different scales. After obtaining the multiscale graph models, we use a node feature optimization network (NFON) to extract and optimize the node features of the graphs in the spatial and frequency domains, and utilize CA to fuse them, thereby obtaining deep-level graph node features. Finally, we use a GNO to solve the optimized node features. NFO-GNO achieves superior solving performance compared to the baselines on four standard benchmarks covering both solid mechanics and fluid dynamics simulations. Notably, NFO-GNO maintains better performance with limited training samples and low-resolution training data, reducing data requirements and making it more adaptable to scenarios where high-quality datasets are difficult to obtain.

Sea surface temperature (SST) prediction is crucial for understanding global climate and marine ecosystems, and its anomalies can lead to extreme weather events. SST exhibits complex non-stationary over natural spatio-temporal processes. However, most of the existing deep learning methods for SST prediction only extract non-stationary features through the simple state transitions of classic CNNs or RNNs, which are too simplistic to capture higher-order non-stationary trends in complex SST sequences. Therefore, we propose a DSNet_SST network, aiming to enhance the extraction of non-stationary information from the spatio-temporal SST evolution. It incorporates two parallel modules: one for capturing high-order temporal non-stationarity based on stacked memory in memory (MIM) blocks and the other one for extracting spatial correlation non-stationarity by multiscale difference operation. A third module adaptively integrates these features to improve the accuracy and stability of SST prediction. The experimental results, relying on the OISST data obtained from both remote sensing satellites and in situ platforms, demonstrate the advantages of the proposed DSNet_SST over baseline methods in terms of SST prediction accuracy.

The widespread application of time-series data has raised higher demands for the accuracy and robustness of anomaly detection. Traditional anomaly detection methods rely on space-frequency perception techniques, expanding the receptive field through frequency-domain transformations. However, the feature confusion between amplitude and phase spectra limits detection performance. To address this issue, we propose a time-series anomaly detection method based on frequency-domain decoupling and correction. The method incorporates two core designs: amplitude-phase decoupling and spectral centroid correction. By separating the complex space in the frequency domain, we independently model amplitude and phase information to avoid feature confusion. Subsequently, the spectral centroid is introduced as a basis for frequency-domain correction to measure the energy distribution of frequency components, adjusting the decoupled amplitude and phase components. This enables the spectrum to more accurately reflect signal characteristics and reduces noise interference. The proposed method provides a finer-grained frequency-domain perspective, enhances the perception of frequency-domain information, and significantly improves the accuracy and robustness of anomaly detection.

The Remote Sensing Copy-Move Question Answering (RSCMQA) task focuses on interpreting complex tampering scenarios and inferring the relationships between objects. Currently, publicly available datasets often use randomly generated tampered images, which lack spatial logic and do not meet the practical needs of defense security and land resource monitoring. To address this, we propose a high-quality manually annotated RSCMQA dataset, Real-RSCM, which provides more realistic evaluation metrics for the identification and understanding of remote sensing image tampering. The tampered images in the Real-RSCM dataset are subtle, authentic, and challenging, posing significant difficulties for model discrimination capabilities. To overcome these challenges, we introduce a multimodal gated mixture of experts model (CM-MMoE), which guides multi-expert models to discern tampered information in images through multi-level visual semantics and textual joint modeling. Extensive experiments demonstrate that CM-MMoE provides a stronger benchmark for the RSCMQA task compared to general VQA and CMQA models. Our dataset and code are available at https://github.com/shenyedepisa/CM-MMoE.

The field of cross-modal retrieval aims to construct a shared representation space for samples from multiple modalities, typically within the vision and language domains. Deep hashing, with its high computational efficiency and low storage costs, has emerged as a central focus in this field and has garnered significant attention in recent research. However, current hash retrieval, concentrating on deterministic methods, struggles to effectively capture semantically ambiguous correspondences between cross-modal samples, where heterogeneous data have complex-semantic many-to-many relationships in the latent space. To address this limitation, we propose a novel Deep Probabilistic Binary Embedding (DPBE) framework, designed to generate discriminative, modality-invariant hash codes that facilitate accurate and reliable cross-modal retrieval. In contrast to contemporary probabilistic methods, we focus on optimizing hash networks to learn more accurate binary embeddings by using the learning mode of probabilistic embeddings. We introduce the first Bayesian encoder for hash learning, which employs Laplace Approximation to model a distribution over network weights. Extensive experimental results demonstrate that our approach not only outperforms deterministic methods in retrieval performance but also provides uncertainty estimates, enhancing the interpretability of the embeddings. The corresponding code is available at https://github.com/QinLab-WFU/DPBE.

Transformer architecture has driven significant advancements in deep hashing, establishing itself as a dominant framework for large-scale retrieval and storage applications. However, existing Transformer-based deep hashing methods typically employ unvarying feature transformations across all images, limiting their adaptability to diverse visual patterns. This rigidity restricts the model's capacity to learn both highly distinctive and generalizable discrete representations, posing challenges for retrieval in open-world scenarios. To overcome this challenge, we propose a novel Factorized Transformer Hashing (FTH) framework, which introduces a factorized transformer to enhance the generalization and discriminative power of hash codes. Specifically, we decompose the Multi-Head Self-Attention (MHSA) and Multi-Layer Perceptron (MLP) blocks into multiple sub-blocks, forming a transformer factorization scheme that captures diverse feature characteristics through independent sub-blocks. Furthermore, we develop an adaptive selection strategy, leveraging a set of learnable selectors with the Softmax function, to dynamically route each image to the most appropriate sub-block for processing. Extensive experiments on three benchmark datasets demonstrate that the proposed FTH framework significantly outperforms state-of-the-art baselines in both image hashing and zero-shot hashing tasks. Source code is available at https://github.com/QinLab-WFU/FTH.

Sea surface temperature (SST) is a crucial metric in marine science, playing a pivotal role in forecasting and analyzing changes in the marine environment. However, remote sensing technologies often encounter issues where SST images are obscured by clouds, leading to data loss, thereby impacting marine environment prediction efficacy. Although many deep learning methods currently exist for reconstructing SST images, most focus on handling this task within the image domain, making it challenging to adapt to the chaotic nature of ocean systems. Additionally, most methods only model at a single scale, which limits their ability to effectively capture the complex multi-scale features in SST data. Therefore, this study proposes MSF_FNO, an image completion method based on Multi-Scale Fourier Fusion Neural Operator. MSF_FNO integrates multi-scale feature fusion and frequency domain neural operator technology to effectively overcome the limitations of single-scale feature processing and image domain reconstruction in existing methods. This approach not only captures SST frequency domain information and extracts structured features of SST images but also extracts critical features across multiple scales, ensuring global consistency and detailed features in reconstruction results. Experiments on the National Satellite Ocean Application Service (NSOAS) datasets demonstrate that MSF_FNO outperforms state-of-the-art (SOTA) methods in terms of reconstruction quality and robustness.

By the preferable efficiency in storage and computation, deep cross-modal has gained much attention in large-scale multimedia retrieval. Current deep hashing employs the probability outputs of the likelihood function, i.e., Sigmoid or Cauchy, to quantify the semantic similarity between samples in a common Hamming space. However, the inherent weakness of the Sigmoid likelihood function or the Cauchy likelihood function in gradient optimization leads to hashing models failing to exactly describe the hamming ball, which indicates the absolute semantic boundary among classes, thereby giving the high neighborhood ambiguity. In this paper, with the analysis of the likelihood function from the perspective of similarity metric learning, the novel Deep Discriminative Boundary Hashing framework (DDBH) is proposed to learn the discriminative embedding space that separates neighbors and non-neighbors well. Specifically, by introducing the remapping strategy and the base-point adaptive selection, the boundary-preserving loss based on the adjustable likelihood function is proposed to project data points with small gradients to regions with large gradients and give larger gradients for hard samples, facilitating better separation among classes. Meanwhile, to learn class-dependent binary codes, the class-wise quantization loss is designed to heuristically transfer class-wise prior knowledge to the binary quantization, significantly improving the discriminative capability of compact discrete codes. Comprehensive experiments on three benchmark datasets show that our proposed DDBH framework outperforms other representative deep cross-modal hashing. The corresponding code is available at https://github.com/QinLab-WFU/DDBH.

Accurate prediction of Sea Surface Temperature (SST) plays a crucial role in climate research, resource development, marine disaster prevention, and environmental protection. Traditional numerical models, while demonstrating excellent predictive accuracy, heavily rely on precise initial and boundary conditions. In contrast, data-driven approaches compensate for these shortcomings with their flexibility and lower computational costs; however, the lack of physical constraints and interpretability limits their effectiveness. Therefore, enhancing model efficiency and interpretability while ensuring accuracy has become a key challenge in current research. This paper proposes a coupled dual-stream SST prediction model that integrates numerical simulation and data-driven methods to improve predictive accuracy and physical consistency. The model includes a bidirectional information compensation module, a physical equation constraint module, and an anomaly compensation module. First, the bidirectional information compensation module constructs physical information flow and complex process flow using Conv-LSTM networks and designs an interaction mechanism between the two streams to enhance the model’s ability to capture complex processes that cannot be explained by partial differential equations. Next, the physical equation constraint module employs data assimilation techniques based on partial differential equations to simulate the dynamics of energy transfer in fluids, ensuring that the prediction results adhere to physical laws. Finally, the anomaly compensation module integrates the prediction results of physical processes and complex processes to output the final SST prediction. Experimental results demonstrate that, compared to existing methods, this model achieves superior predictive accuracy across multiple datasets.

Marine environmental forecasting presents a significant challenge in geoscience and remote sensing applications, particularly when modeling the complex multi-scale dynamics evident in satellite-derived sea surface temperature (SST) data. While recent deep learning approaches have shown promise in processing remote sensing data, they often fail to effectively capture the rich semantic information inherent in multi-scale oceanic phenomena, resulting in suboptimal prediction performance. To address this limitation, we propose MS-LIP, a novel neural network framework that leverages large language models (LLMs) to integrate multi-scale semantic information from satellite data for enhanced marine element prediction, with primary focus on SST forecasting. Our key innovation, the Multi-Scale Semantic Information Extraction Module (MSIE), employs cross-modal attention mechanisms to dynamically align LLM-derived semantic representations with remotely sensed spatiotemporal features at multiple scales, enabling comprehensive capture of thermal patterns from localized anomalies to basin-wide circulations. Additionally, we introduce the Amplitude-Phase Decoupling Interaction Module (APDI), which performs frequency-domain analysis specifically designed for satellite-derived SST data, extracting deeper insights from thermal patterns that conventional methods typically overlook. These components are integrated into our Domain-Specific Fine-Tuning for Pre-trained Large Language Models (DFPM) framework, which incorporates an efficient Low-Rank Adaptation with Mixture of Experts (LoRA-MoE) strategy tailored to marine remote sensing applications. Experimental validation using satellite-derived NOAA OISST data demonstrates state-of-the-art SST prediction performance, with error reductions of 7.12%, 5.88%, and 2.40% for temperature ranges around 15°C, 20°C, and 25°C, respectively. Further experiments on ERA5 reanalysis data for significant wave height (SWH) prediction demonstrate the framework’s generalizability to other marine elements, highlighting its versatility and potential for broad application in marine environmental forecasting.

Ocean data has strong spatiotemporal correlations and is also characterized by multi-source, decentralized distribution, presenting significant challenges for distributed collaboration. To address these issues, in this paper, we propose DGLN, a distributed GCN-based prediction architecture that employs a decentralized training paradigm comprising local and distributed systems. The local system is designed to capture scale-specific features of marine data across different processors, while the distributed system learns and coordinates global cross-timescale features. The graph learning module in the distributed system integrates graph features from other processors without transmitting raw data, thereby ensuring local data security. To accelerate training and enhance model performance, a joint learning module is composed by cross-timescale computation with a new global weighted consensus algorithm, which is able to adaptively adjust the weights of different scales. Experimental results show that DGLN is able to improve the local prediction accuracy and capture cross-timescale features effectively while improving the training efficiency.

Recently, remote sensing text-image retrieval has gained increasing attention among researchers due to its capability to provide abundant, inclusive, and multi-perspective information. To extract the salient representation of the two modalities and realize information alignment, existing methods usually apply convolution calculation, matrix multiplication, and attention mechanisms to model features in the spatial domain. However, unlike natural images, remote sensing data contain a considerable amount of noise, and the spatial domain calculation methods will lead to information smoothing and noise exchange, ultimately reducing the robustness of feature representation. In this article, a Frequency and Spatial-domain Saliency Network (FSSN) is proposed by extracting saliency features in both frequency and spatial domains to further enhance the efficiency of retrieval networks. Specifically, the FSSN first designs a Frequency-domain Intra-modal Low-pass Filter (FILF) by using the Fourier Transform to convert the spatial domain representation into the frequency domain representation, and leverage low-pass filtering to filter out the noise information of the image. Afterwards, the Space-domain Inter-modal Boundary-based Saliency module (SIBS) is devised, which fully utilizes the positive and negative boundaries and draws an innovative assessment mechanism to automatically recognize the effective regions and words without noise shifting in the spatial domain. Finally, the Frequency and Spatial-domain Saliency Fusion (FSSF) is designed to realize the effective integration of features obtained from the frequency and spatial domain according to the consistency of the identity masks. Quantitative and qualitative experiments are implemented on four remote sensing benchmarks to showcase the notable effectiveness achieved by the joint modeling network of multiple domains. Specifically, the proposed FSSN outperforms the best model RemoteClip by 7.58% of the mR evaluation matrix on the UCM dataset.

In recent years, cross-modal text-image retrieval in remote sensing has gained prominence as a research focus due to its potential to provide abundant, inclusive, and multi-perspective information. However, existing methods usually focus on salient features, but these salient features cannot describe the image or text completely, resulting in the loss of some important details and discriminable information. In this article, a Whole Semantic Sparse Coding Network (WSSCN) is proposed for remote sensing image-text retrieval to build a complete and reliable features description for further improving the performance of the retrieval model. Specifically, the WSSCN first designs a Whole Semantic Sparse Representation Coding (WSSRC) module by constructing a robust semantic library to transform the dense features matrix of the image and text into a whole semantic sparse matrix that enables multiple semantic decoupling and leads to a more precise and detailed features expression. Afterwards, the Intra- and Inter-Modal Consistency (IIMC) module is devised to improve the intra-modal and inter-modal consistency of the whole semantic sparse representation from different models. Finally, the Salient and Whole Semantic Adaptive Learning (SWSAL) loss is proposed to focus on salient information or whole semantic information by calculating an adjusted parameter. Quantitative and qualitative experiments are performed on four extensive datasets for cross-modal retrieval in remote sensing to showcase the notable effectiveness achieved by implementing a whole semantic sparse coding network.

The Scientific Visualization Images (SVI) of Sea Surface Temperature (SST) play a pivotal role as visual resources for investigating oceanographic processes. However, they are often plagued by extensive data gaps due to objective factors like cloud cover. Additionally, their content deviates significantly from ordinary images, posing challenges for conventional completion techniques. Given the intricate nature of marine systems, completing the visualization of sea surface temperature presents several challenges. Firstly, predicting missing segments relies not only on prominent patterns but also on subtle anomalies, which are often overlooked by methods focused on extracting prominent features. Secondly, these images exhibit chaos and lack clear semantics, making it difficult for methods primarily focused on semantic extraction to effectively complete them. To address these challenges, this study presents a novel method named the Inpainting Fourier Neural Network for SVI (SVIFNN). This approach employs a twin-stream architecture to highlight both significant stability and non-significant anomalies. Notably, it incorporates a "reverse attention mechanism" in the non-significant anomalies extraction stream to preserve unconventional information. Furthermore, by cascading Fourier neural operator (FNO), it leverages frequency domain characteristics to mitigate spatial chaos. Through a frequency domain feature extraction module, it achieves an adaptive fusion of significant stability and nonsignificant anomalies. Experiment results demonstrate SVIFNN’s superiority over State-Of-The-Art (SOTA) methods, particularly under a 68% missing rate condition. Significant improvements are observed inR2(18.1%, 19.3%, and 21.8%) and reductions inRMSE(22.6%, 28.8%, and 23.8%) across different Noise-to-Signal (N/S) ratios of 0.1, 0.2, and 0.3, respectively, underscoring SVIFNN’s robustness in handling SVIs extensive data gaps. Adequate ablation experiments further validate the effectiveness of the proposed non-significant anomalies extraction stream and frequency domain operators, with the latter demonstrating superior performance for scientific visualization images compared to traditional spatial domain CNN and ViT operators.

Benefiting from the advantages of low storage cost and high retrieval efficiency, hash learning could significantly speed up large-scale cross-modal retrieval. Based on the prior annotations, most of the available cross-modal hashing usually introduces the margin-based constraint to generate different boundaries for each class in the inference phase, optimizing the model. However, these obtained label-guided penalty boundaries may differ from the primitive semantic relationships between heterogeneous modalities, impairing retrieval performance. Besides, the margin-based constraint is too weak to penalize the classes with low intra-class variances or inter-class correlations, which struggle to learn high-quality embeddings. In this paper, we propose a novel Deep Semantic-consistent Penalizing Hashing framework (DScPH) to learn the consistent penalizing fields for all classes, achieving accurate and efficient cross-modal retrieval. Specifically, by exploring unbalanced intra-class and inter-class correlations, the consistent penalizing loss is introduced into cross-modal retrieval to learn the consistency decision boundaries across classes. During training, the dice-like optimization strategy is developed to balance the pulling penalizing elements and pushing penalizing elements, facilitating the model convergence. Besides, based on the invariance of similarity measures under orthogonal transformations, the alternative quantization is proposed to minimize the errors between the learned continuous embeddings and binary discretization, maintaining the consistency of semantic relationships after performing binary projection. Extensive experiments are conducted on three benchmark datasets, and the comprehensive results validate the efficacy of our proposed DScPH framework, which outperforms the current mainstream deep cross-modal hashing algorithms.

The detection of image tampering, specifically copy detection, is an important problem in many domains such as military, media, and public opinion outlets. Effective means to detect such tampering is crucial in controlling the dissemination of false information. However, a major challenge in achieving high detection accuracy lies in the variability of the scale of the copied targets. To tackle this problem, we introduce an all-encompassing methodology called Cross-Scale Modeling and Alternating Refinement (CANet) to detect the genuine source and tampered region at the pixel level. CANet consists of three modules: the Cross-Scale Similar Region Detection (CS) module, the Edge-Supervised Tamper Region Detection (ET) module, and the Alternating Refinement (AR) module. The CS module extracts coarse similar region features by cross-scale correlation modeling, which can alleviate the scale gap between the source and tampered region. The obtained coarse similar region feature is refined by the AR module, in which we introduce the source and the tampered region as the auxiliary information and employ a two-stage process that sequentially models their global feature representations. The tampered region used in the AR module is obtained from the ET module using edge supervision with a salient edge selection scheme, and the source region is generated by the implicit modeling. We conducted experiments on the USC-ISI, CASIA v2.0, CoMoFoD, and MICC-F220 datasets separately. Results show that our method outperforms the state-of-the-art.

Cross-modality based on remote sensing (RS) text-image retrieval has gained increasing attention in recent years due to its ability to leverage the rich semantics of images and the understandability of text to provide a more comprehensive description. Existing cross-modal retrieval methods typically apply self-attention or cross-attention mechanisms to identify important information in RS data, but they ignore the multi-view perception characteristic of geographical space in RS images. As a result, these retrieval models fail to locate the correct perspective in images according to the query text, ultimately leading to incorrect matching. In this work, a Cross-modal Progressive Perspective Matching Network (CPPMN) is proposed for remote sensing image-text retrieval by establishing a progressive perspective matching mechanism and semantic alignment to further improve the performance of the retrieval model. Specifically, the CPPMN framework consists of three core modules: the Compensation Network for Full Perspective Modeling (CN_FPM), the Graph Transformation for Individual Perspective Modeling (GT_IPM), and the Cascaded Transformer for Cross-modal Semantic Alignment (CT_CSA). The CN_FPM module utilizes all positive text samples as supervision signals to guide the feature extraction training process, aiming to capture full perspective information from images. Subsequently, the GT_IPM module transforms implicit-perspective feature representations into explicit-perspective cross-modal relationship graphs. This transformation enables the identification of specific perspective locations within the image according to the query sentence by analyzing graph density and connectivity. Finally, the CT_CSA module comprises a cascaded Transformer network that aligns features at the semantic level between cross-modal data The quantitative and qualitative experiments are conducted on four large-scale remote sensing cross-modal retrieval datasets to demonstrate the significant performance of adopting the progressive perspective matching mechanism and semantic alignment strategy.

In the predictive maintenance of modern industries, accurate fault diagnosis under complex conditions is now a major research focus. Recent research has demonstrated the effectiveness of deep learning in advancing bearing fault diagnosis. However, due to the scarcity of industrial failure data, achieving robust generalization in complex working conditions remains a challenge. To address this, we propose the causal disentanglement-based hidden Markov model (CDHM), which is designed to recognize the underlying causality in bearing vibration signals, capturing essential fault patterns for a more accurate and generalizable fault representation. Compared to signal-processing methods, deep learning approaches bypass the complex signal analysis, yet overlook the significance of signal theories in precise fault diagnosis. Nevertheless, the bearing vibration mechanism sheds light on the fact that the vibration induced by a certain type of fault has a consistent pattern across different system conditions, while the fault-irrelevant vibration such as noise and interference varies. Therefore, the CDHM constructs a time-series structural causal model (SCM), offering a new perspective on the interconnections of bearing vibration signals. Based on the SCM, a hidden Markovian variational autoencoder (VAE) is designed to progressively disentangle the vibration signal into two parts: a fault-relevant representation capturing essential bearing fault characteristics, and a fault-irrelevant representation capturing system and environmental interference. While unsupervised causal disentanglement typically presents optimization challenges, the CDHM benefits from cross-domain fault diagnosis tasks by leveraging the cross-domain consistency of the fault-relevant representation and the domain sensitivity of the fault-irrelevant representation. This design aligns the optimization objectives of causal disentanglement learning and cross-domain transfer learning, enabling mutually reinforcing optimization and ensuring robust generalization across diverse operating conditions. We validate the CDHM through experiments on the Case Western Reserve University (CWRU), Intelligent Maintenance System (IMS), and Paderborn University (PU) datasets, demonstrating its strong potential for industrial applications.

Cloth-changing person re-identification (ReID) is a newly emerging research topic that aims to retrieve pedestrians whose clothes are changed. Since the human appearance with different clothes exhibits large variations, it is very difficult for existing approaches to extract discriminative and robust feature representations. Current works mainly focus on body shape or contour sketches, but the human semantic information and the potential consistency of pedestrian features before and after changing clothes are not fully explored or are ignored. To solve these issues, in this work, a novel semantic-aware attention and visual shielding network for cloth-changing person ReID (abbreviated as SAVS) is proposed where the key idea is to shield clues related to the appearance of clothes and only focus on visual semantic information that is not sensitive to view/posture changes. Specifically, a visual semantic encoder is first employed to locate the human body and clothing regions based on human semantic segmentation information. Then, a human semantic attention (HSA) module is proposed to highlight the human semantic information and reweight the visual feature map. In addition, a visual clothes shielding (VCS) module is also designed to extract a more robust feature representation for the cloth-changing task by covering the clothing regions and focusing the model on the visual semantic information unrelated to the clothes. Most importantly, these two modules are jointly explored in an end-to-end unified framework. Extensive experiments demonstrate that the proposed method can significantly outperform state-of-the-art methods, and more robust features can be extracted for cloth-changing persons. Compared with multibiometric unified network (MBUNet) (published in TIP2023), this method can achieve improvements of 17.5% (30.9%) and 8.5% (10.4%) on the LTCC and Celeb-reID datasets in terms of mean average precision (mAP) (rank-1), respectively. When compared with the Swin Transformer (Swin-T), the improvements can reach 28.6% (17.3%), 22.5% (10.0%), 19.5% (10.2%), and 8.6% (10.1%) on the PRCC, LTCC, Celeb, and NKUP datasets in terms of rank-1 (mAP), respectively.

Visual question answering (VQA) is a challenging task that requires models to understand both visual and linguistic inputs and produce accurate answers. However, VQA models often exploit biases in datasets to make predictions rather than reasoning based on the inputs. Prior approaches to debiasing have suggested the implementation of a supplementary model, deliberately designed to exhibit bias, which subsequently informs the training of a resilient target model. Nevertheless, such techniques merely quantify the model’s divergence based on the statistical distribution of labels within the training dataset or in relation to unimodal branches. In this work, we propose a novel method of generating bias from the target model itself, called LEGO, which aims to combat the language guidance bias. Specifically, LEGO framework employs a generative network that assimilates the biases inherent in the target model by integrating adversarial goals with the principles of knowledge distillation. Then, we use a debiased contrastive learning strategy to model the language guidance bias of caption and question. In the process of modeling, in order to obtain robust semantic coreference, the multimodal representations of two semantic granularity are modeled by mutual information fusion and contrast learning difference modeling. We evaluate our method on various VQA-biased datasets, including VQA-CP2, GQA-OOD, and RSICD, and show that it outperforms similar methods.

Large language models have demonstrated strong performance in recent years, but the high cost of training drives the need for efficient methods to compress dataset sizes. We propose TED pruning, a method that addresses the challenge of overfitting under high pruning ratios by quantifying the model’s ability to improve performance on pruned data while fitting retained data, known as Internal Generalization (IG). TED uses an optimization objective based on Internal Generalization Distance (IGD), measuring changes in IG before and after pruning to align with true generalization performance and achieve implicit regularization. The IGD optimization objective was verified to allow the model to achieve the smallest upper bound on generalization error. The impact of small mask fluctuations on IG is studied through masks and Taylor approximation, and fast estimation of IGD is enabled. In analyzing continuous training dynamics, the prior effect of IGD is validated, and a progressive pruning strategy is proposed. Experiments on image classification, natural language understanding, and large language model fine-tuning show TED achieves lossless performance with 60-70% of the data. Upon acceptance, our code will be made publicly available.

Ocean data has sheer volume, distributed sources, and spatiotemporal correlations, resulting in difficulties for distributed information collaboration. To tackle this problem, we propose a decentralized learning framework consisting of a spatial correlation modeling scheme based on k-shot data sampling and an asynchronous communication scheme for decentralized network establishing and information exchange. The proposed framework can handle the sparse communication scenario and improve the performance of the local learning models.

The sampling in the Earth system poses challenges to meeting the assumption of independent and identically distributed samples. However, existing deep neural network forecasting models inadequately address this issue, impacting the model's generalization and stability. To tackle this problem, this paper introduces the Non-Independent Sample Bias Elimination Module and the Non-Identically Distributed Prototype Modeling Module within the classical deep neural network forecasting framework. The effectiveness of the proposed approach is validated through forecasting experiments on sea surface temperature and sea surface height in the Chinese sea region.

Sea surface temperature (SST), a crucial indicator widely applied in marine-related fields, is often observed by extensive cloud cover in satellite observations, resulting in significant data gaps. While deep learning approaches offer solutions for reconstructing SST images using ample historical data, the majority of current methods overlook the context constraints inherent in oceanographic data, such as sequential correlations and global consistency. This oversight leads to non-robust completion results in the presence of extensive data gaps. This paper proposes a robust SST reconstruction network guided by sequential correlations and global consistency, designed to handle the task of extensive missing data recovery in SST under large-scale cloud cover, namely CCG_RRN. Extensive comparison experiments and visualization results affirm the effectiveness and robustness of CCG_RRN compared with the State-Of-The-Art (SOTA) methods on the public NSOAS SST datasets.

Neural network pruning has become a mainstream approach for reducing network complexity and accelerating inference speed. Pre-training pruning, achieved by removing redundant connections and neurons before model training, effectively reduces network size, lowering computational and storage costs. Existing pre-training pruning methods face the risk of layer collapse at high compression rates, while generalization performance needs improvement at typical compression rates. We propose the concept of kernel chains in this paper, finely characterizing inter-layer alignment features of pruned models at different pruning rates. By constructing connection strength through the derivative of the current layer’s output with respect to the weights of the previous layer, we reflect the synchronous changes between weights and their corresponding upstream and downstream kernel chains. We utilize synchronous information to assess the importance of weights, making the acquisition of synchronous information possible based on the kernel chain and connection strength. Based on this, we introduce a novel general pruning method that ensures network operability at high compression rates and maximizes generalization improvement at typical compression rates. Extensive experiments demonstrate that our method achieves state-of-the-art performance when using VGG and ResNet networks on CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets. The code is publicly available at https://github.com/hunkoufanchi777/SynCW

Large-scale Transformer models (TM) have demonstrated outstanding performance across various tasks. However, their considerable parameter size restricts their applicability, particularly on mobile devices. Due to the dynamic and intricate nature of gradients on TM compared to Convolutional Neural Networks, commonly used pruning methods tend to retain weights with larger gradient noise. This results in pruned models that are sensitive to sparsity and datasets, exhibiting suboptimal performance. Symbolic Descent (SD) is a general approach for training and fine-tuning TM. In this paper, we attempt to describe the noisy batch gradient sequences on TM through the cumulative process of SD. We utilize this design to dynamically assess the importance scores of weights.SEVEN is introduced by us, which particularly favors weights with consistently high sensitivity, i.e., weights with small gradient noise. These weights are tended to be preserved by SEVEN. Extensive experiments on various TM in natural language, question-answering, and image classification domains are conducted to validate the effectiveness of SEVEN. The results demonstrate significant improvements of SEVEN in multiple pruning scenarios and across different sparsity levels. Additionally, SEVEN exhibits robust performance under various fine-tuning strategies. The code is publicly available at https://github.com/xiaojinying/SEVEN.

Pruning before training enables the deployment of neural networks on smart devices. By retaining weights conducive to generalization, pruned networks can be accommodated on resource-constrained smart devices. It is commonly held that the distance on weight norms between the initialized and the fully-trained networks correlates with generalization performance. However, as we have uncovered, inconsistency between this metric and generalization during training processes, which poses an obstacle to determine the pruned structures on smart devices in advance. In this paper, we introduce the concept of the learning gap, emphasizing its accurate correlation with generalization. Experiments show that the learning gap, in the form of feature maps from the penultimate layer of networks, aligns with variations of generalization performance. We propose a novel learning framework, LNPT, which enables mature networks on the cloud to provide online guidance for network pruning and learning on smart devices with unlabeled data. Our results demonstrate the superiority of this approach over supervised training.

Unsupervised deep hashing has demonstrated significant advancements with the development of contrastive learning. However, most of previous methods have been hindered by insufficient similarity mining using global-only image representations. This has led to interference from background or non-interest objects during similarity reconstruction and contrastive learning. To address this limitation, we propose a novel unsupervised deep hashing framework named Fine-grained Similarity-preserving Contrastive learning Hashing (FSCH), which explores fine-grained semantic similarity among different images and their augmented views more comprehensively. It mainly comprises two modules: the global-local fine-grained similarity consistency preservation module and the local fine-grained similarity contrast preservation module. Specifically, we reconstruct local pairwise similarity structures by matching fine-grained patches, in conjunction with global similarity structures based on global hash codes cosine similarity, to generate hash codes with the ability to preserve global-local similarity consistency. Moreover, the preservation of local fine-grained similarity among augmented views is accomplished through the common regional features mutual representation between patches, then we enhance the discriminability of hash codes by mitigating the potential features difference during contrastive learning. Experimental results on four benchmark datasets demonstrate that our FSCH achieves an excellent retrieval performance compared to state-of-the-art unsupervised hashing methods.

Hyperspectral unmixing is a crucial task for identifying the constituent materials and their respective distributions in a scene. Utilizing known spectral libraries as prior information, semi-blind unmixing methods (also known as spectral-library-based methods) have been proven advantageous over unblind and blind approaches. However, such methods encounter two main challenges: difficulty in handling large-scale spectral libraries and vulnerability to variabilities stemming from differences between the underlying signatures and those in the spectral library. To address these challenges, a novel approach named DiffUn, based on a diffusion model, is proposed in this article for semi-blind hyperspectral unmixing. DiffUn considers hyperspectral unmixing as a sampling process from a posterior distribution, where the prior distribution is learned from a spectral library, and the likelihood distribution is estimated from the observed data by the linear spectral mixture model. Specifically, the approach first learns the spectral prior distribution from a spectral library through an unconditional diffusion model, then integrates this prior knowledge into the reverse process of the diffusion model, and finally samples the underlying endmembers and corresponding abundances from the posterior distribution. Since spectral prior distribution estimation is not sensitive to library size, DiffUn exhibits superior unmixing performance even in a large-scale library. Furthermore, DiffUn permits sampling spectral signatures from a continuous probabilistic distribution, whereas conventional semi-blind unmixing methods only allow endmembers selected from the library, which is a discrete space. Thus, DiffUn shows greater robustness to spectral variations. Experimental results on synthetic and real-world datasets demonstrate DiffUn outperforming the state-of-the-art semi-blind unmixing methods. The code is available at https://github.com/Dmsw/DiffUn.git.

In recent years, there has been a growing demand for the colorization of remote sensing images due to their inherent limitations caused by remote sensors, such as hazy or noisy atmospheric conditions. These factors result in the captured images needing to be clarified. Compared to ordinary images, remote sensing images present unique challenges in color recovery due to their imbalanced spatial distribution of objects. In this article, we propose a novel bidirectional layout-semantic-pixel joint decoupling and embedding network (BDEnet) following the idea of human painting to generate highly saturated color images with strong spatial consistency and object salience. The proposed BDEnet model emulates the process of human painting through a step-by-step approach. It begins by determining the overall tone of a large macroscopic region and progressively refining the local color based on this initial assessment. Specifically, BDEnet incorporates finer-grained semantics and pixel color information into a colored layout that represents a wide range of continuous areas, thereby accomplishing the colorization task. The BDEnet model operates at three scales, namely the layout (macro), semantic (medium), and pixel (micro) scales. It comprises three key modules: the multiscale feature decoupling (MFD) module, the layout-semantic-pixel multigranularity learning (MGL) module, and the semantic-pixel embedding (SPE) module. MFD module effectively reduces redundant noise from the semantic and layout scales by employing scale decoupling. This process ensures the extraction of efficient features essential for MGL. In the MGL module, three branches with different scales are employed to achieve layout division, semantic segmentation, and pixel coloring. To address the issue of insufficient category label guidance in layouts, we propose a novel approach called similar semantic merging (SSM) using a weakly supervised scheme to accomplish layout division. Finally, the SPE module incorporates stable semantic and pixel information into the layout features. This integration results in the generation of color images that exhibit strong spatial consistency, emphasize object salience, and possess high color saturation.

Remote sensing visual grounding (RSVG) represents a pivotal task aimed at pinpointing regions highly relevant to textual descriptions by parsing remote sensing image content. Existing RSVG methodologies focus on multimodal semantic integration on specific datasets, overlooking the intricate high-dimensional information present in remote sensing images, where color, scale, and semantics are tightly coupled. Consequently, these approaches exhibit limitations in handling specific receptive fields or preserving structural information within remote sensing images, leading to insufficient localization precision. To address this issue, this article introduces a strategy that incorporates spatial frequency information, leveraging Fourier transforms to capture global structured information from remote sensing data. This is followed by a progressive aggregation across spatial, spectral, and linguistic modalities to achieve robust semantic coreference. The primary contributions of this article are as follows: First, a spatial-frequency fusion strategy based on linguistic query refinement is proposed, which enhances visual grounding performance significantly by extracting spectral features with potent spatial perception capabilities through the expansion of the spectral receptive field. Second, a frequency-guided spatial (FGS) module is designed, utilizing amplitude and phase-structured spectral features to augment spatial representation capabilities further. Lastly, a query-aware original attention (QOA) mechanism is developed, facilitating deep integration of spatial and spectral information under linguistic guidance. Extensive experimentation on the RSVGD dataset validates the efficacy of the proposed approach, demonstrating superior performance when compared to state-of-the-art methods.

Sea surface temperature (SST) serves as a critical indicator of oceanic environmental changes. However, the uninterrupted observation of vast oceanic areas via remote sensing is frequently impeded by cloud cover, resulting in persistent data gaps. Consequently, the completion of SST data emerges as an essential technique. Recently, the utilization of deep neural networks (DNNs), particularly generative models, has shown promising results by effectively leveraging historical data for training. However, these approaches often concentrate on spatial domain features, neglecting the inherent chaotic nature of the ocean as a complex system, thus failing to accurately reflect the complexity of oceanic processes. Therefore, this study introduces a novel approach to SST completion focusing on frequency domain features, utilizing Fourier neural operators. We propose an innovative Data INpainting Fourier Neural Network (DINFNN), for data completion to facilitate feature learning in complex oceanic systems. Our method employs a triple-stream neural network to capture periodic steady-state features, adjacent temporal features, and current context. By integrating high-pass and low-pass filters and dynamically combining them, we adaptively extract essential frequency domain features for completion. Particularly, a two-cascaded fusion module is utilized dynamically to amalgamate these distinct attributes to form a composite feature set, encompassing both steady and dynamic frequency domain elements, aimed at achieving a comprehensive SST field reconstruction. In our experimental evaluations, our method demonstrates significant improvements under various conditions. Specifically, when the cover ratio is 68% and the Noise-to-Signal (N/S) is set to 0.1, 0.2, and 0.3, we observe enhancements of 11.2%, 10.8%, and 16.2% in R-squared ($R^{2}$), respectively. Furthermore, corresponding reductions of 7.2%, 13.1%, and 7.3% in root mean square error (RMSE) are achieved, respectively. Moreover, comprehensive ablation experiments confirm the effectiveness of each component within our method and emphasize the superiority of DINFNN over conventional operators.

Video question answering (VideoQA) requires the ability of comprehensively understanding visual contents in videos. Existing VideoQA models mainly focus on scenarios involving a single event with simple object interactions and leave event-centric scenarios involving multiple events with dynamically complex object interactions largely unexplored. These conventional VideoQA models are usually based on features extracted from the global visual signals, making it difficult to capture the object-level and event-level semantics. Although there exists a recent work utilizing a static spatio-temporal graph to explicitly model object interactions in videos, it ignores the dynamic impact of questions for graph construction and fails to exploit the implicit event-level semantic clues in questions. To overcome these limitations, we propose a Self-supervised Dynamic Graph Reasoning (SDGraphR) model for video question answering (VideoQA). Our SDGraphR model learns a question-guided spatio-temporal graph that dynamically encodes intra-frame spatial correlations and inter-frame correspondences between objects in the videos. Furthermore, the proposed SDGraphR model discovers event-level cues from questions to conduct self-supervised learning with an auxiliary event recognition task, which in turn helps to improve its VideoQA performances without using any extra annotations. We carry out extensive experiments to validate the substantial improvements of our proposed SDGraphR model over existing baselines.

Due to its low storage cost and high retrieval efficiency, hashing technology is popularly applied in both academia and industry, which provides an interesting solution for cross-modal similarity retrieval. However, most existing supervised cross-modal hashing methods typically view the fixed-level semantic affinity defined by manual labels as supervised signals to guide hash learning, which only represents a small subset of complex semantic relations between multi-modal samples, thus impeding the hash function learning and degrading the obtained hash codes. In the paper, by learning shared hierarchy proxies, a novel deep cross-modal hashing framework, called Deep Hierarchy-aware Proxy Hashing (DHaPH), is proposed to construct the semantic hierarchy in a data-driven manner, thereby capturing the accurate fine-grained semantic relationships and achieving small intra-class scatter and big inter-class scatter. Specifically, by regarding the hierarchical proxies as learnable ancestors, a novel hierarchy-aware proxy loss is designed to model the latent semantic hierarchical structures from different modalities without prior hierarchy knowledge, in which similar samples share the same Lowest Common Ancestor (LCA) and dissimilar points have different LCA. Meanwhile, to adequately capture valuable semantic information from hard pairs, a multi-modal self-paced loss is introduced into cross-modal hashing to reweight multi-modal pairs dynamically, which enables the model to gradually focus on hard pairs while simultaneously learning universal patterns from multi-modal pairs. Extensive experiments on three available benchmark databases demonstrate that our proposed DHaPH framework outperforms the compared baselines with different evaluation metrics. The corresponding code is available athttps://github.com/QinLab-WFU/DHaPH.

Unsupervised domain adaptation (UDA) involves the transfer of knowledge from a labelled source domain to an unlabelled target domain. Recent studies have introduced the concept of intermediate domains to handle significant domain discrepancies between source and target domains. Constructing an appropriate intermediate domain is a crucial step in handling scenarios with substantial domain differences. In this paper, we propose a novel progressive UDA method called adaptive multi-scale intermediate domain via progressive training (AMPT), which has achieved remarkable effectiveness in alleviating large discrepancies between domains. We design a multi-scale similarity metrics module to solve the issue of different scales across different domains by simultaneously computing the pairwise distance between source domain images and the target domain images on multiple scales. Furthermore, we explore a progressive training strategy that facilitates smooth adaptation of the source domain to the target domain by utilizing the intermediate domain. During the progressive training process, considering a positive feedback mechanism, we iteratively leverage task losses and distillation loss through a dynamic threshold. This ensures that the trained intermediate domain branch progressively constrains the target domain branch, and the intermediate domain can be generated dynamically, leading to a smoother gradual adaptation from the source domain to the target domain. Extensive experimental results demonstrate the superiority of our proposed method AMPT on well-known UDA datasets, including Office-Home, Office-31 and DomainNet.

Image-text retrieval in remote sensing aims to provide flexible information for data analysis and application. In recent years, state-of-the-art methods are dedicated to “scale decoupling” and “semantic decoupling” strategies to further enhance the capability of representation. However, these previous approaches focus on either the disentangling scale or semantics but ignore merging these two ideas in a union model, which extremely limits the performance of cross-modal retrieval models. To address these issues, we propose a novel Scale-Semantic Joint Decoupling Network (SSJDN) for remote sensing image-text retrieval. Specifically, we design the Bidirectional Scale Decoupling (BSD) module, which exploits Salience Extraction Map (SEM) and Salience Suppression Map (SSM) units to adaptively extract potential features and suppress cumbersome features at other scales in a bidirectional pattern to yield different scale clues. Besides, we design the Label-supervised Semantic Decoupling (LSD) module by leveraging the category semantic labels as prior knowledge to supervise images and texts probing significant semantic-related information. Finally, we design a Semantic-guided Triple Loss (STL), which adaptively generates a constant to adjust the loss function to improve the probability of matching the same semantic image and text and shorten the convergence time of the retrieval model. Our proposed SSJDN outperforms state-of-the-art approaches in numerical experiments conducted on four benchmark remote sensing datasets.

Data partitioning is the most fundamental procedure before parallelizing complex analysis on very big graphs. As a classical NP-complete problem, graph partitioning usually employs offline or online/streaming heuristics to find approximately optimal solutions. However, they are either heavyweight in space and time overheads or suboptimal in quality measured by workload balance and the number of cutting edges across partitions, both of which cannot scale well with the ever-growing demands of quickly analyzing big graphs. This paper thereby proposes a new vertex partitioner for better scalability. It preserves the lightweight advantage of existing streaming heuristics, and more importantly, fully utilizes the knowledge embedded in the local view when streaming a vertex, which significantly improves the quality. We present a sliding window technique to compensate for the additional memory costs caused by knowledge utilization. Also, a parallel technique with dependency detection optimization is designed to further enhance efficiency. Experiments on a spread of real-world datasets validate that our proposals can achieve overall success in terms of partitioning quality, memory consumption, and runtime efficiency.

Local differential privacy (LDP) has been established as a strong privacy standard for collecting sensitive information from users. Although it has attracted much research attention in recent years, the majority of existing works focus on applying LDP to frequency distribution estimation for each individual value in a discrete domain. This paper concerns the important range queries involving multiple discrete values. Till now, only a few works target this problem. They all rely on the B-ary tree to construct a uniform and hierarchical decomposition, so as to decrease the error when answering large range queries. However, the uniform splitting manner ignores the properties of decomposed sub-domains and processes them equally without preferences, which leads to significant performance penalty.In this paper, we tackle the problem head on: our proposal, privNUD, is a novel domain hierarchical decomposition mechanism. It dynamically decomposes each domain with a tailored granularity into some sub-domains, which sensitively considers the potential chances to answer one range query. The issue of granularity is carefully analyzed for better performance. It also can smartly prune the sub-domains with small frequencies. Besides, an adaptive user allocation technique is designed to dynamically decide the scale of users that are involved in each sub-domain’s frequency estimation. Extensive experiments using real and synthetic datasets demonstrate that privNUD achieves significantly higher result accuracy compared to the up-to-date solutions.

Sea Surface Temperature (SST) is a significant environmental factor indicating marine revolutions, which is popularly applied in the meteorological forecasting and fishing industry. Due to the limited sensing ability and occlusion caused by clouds or ice, it is difficult to obtain complete SST data. Compared to traditional interpolation-based methods which refill missed data only referred to current SST data, inpainting-based methods have been carried out with the advantage of using historical SST images to train Generative adversarial Networks (GAN) by terms of considering SST data reconstruction task as an image inpainting task. However, different from common inpainting tasks constrained by semantics, the SST image is a scientific data visualization image without semantics but physical constraints. To address this problem, this paper proposes a multi-scale inpainting GAN-based neural networks to guarantee the physical constraint and realize reasonable SST image reconstruction. The proposed framework mainly contains two modules including the Average Estimation Module (AEM) to realize a global constraint so as not to generate excessive deviation, and the Multi-scale Anomaly Decouple Module (MSADM) to preserve data specificity of current SST image from well-designed multi-scale and decoupled perspectives. Finally, a post-fusion module concatenates the "average" and "specificity" features together to accomplish our multi-scale physical constraints SST image inpainting task. Sufficient experiments have been carried out to verify the effectiveness and physical consistency compared with prior SOTA methods applied to the public AVHRR Pathfinder SST dataset.

Remote sensing cross-modal image-text retrieval has attracted increasing attention due to its important roles in multiple domains. Existing methods perform salient modeling for the feature that has high relevance between different modalities. However, most works consider the relevance between different modalities but ignore the irrelevance between different modalities, resulting in incomplete modeling of the relevance and irrelevance between different modalities. In this paper, we propose a Relevance and Irrelevance Considered Subspace Mapping Neural Networks (RIR-SMNNs) to simultaneously consider the relevance and irrelevance between different modalities. Specifically, we first utilize Multiscale Image Feature Extraction (MIFE) and Multiscale Text Feature Extraction (MTFE) to extract the multiscale feature of image and text. Then, we perform Local Space Building Module (LSB), which constructs local space that realizes scale alignment. Finally, we perform the Relevance and Irrelevance Local Space Mapping (RIRLSM) to consider the relevance and irrelevance of different modalities in multiple spaces. Experimental results on several remote sensing datasets demonstrate our model outperforms the state-of-the-art approaches.

In recent years, semantic segmentation technology plays an important role in land resource management tasks. However, many classic semantic segmentation methods often fail to obtain satisfactory results for remote sensing images with a large amount of interference information. In order to improve this situation, we propose Semantic Category Balance-Aware Involved Anti-Interference Network(SCBANet). SCBANet has an encoder-decoder structure similar to DeeplabV3+ [1]. On this basis, we propose Clustering Guided Semantic Decoupling Module(CGSDM), Consistency Based Anti-Interference Feature Extraction Module(CAFEM), Relevance Based Anti-Interference Feature Extraction Module(RAFEM) and Optional Decoder Module Based on Semantic Category Balance(ODMSCB) to improve the accuracy of semantic segmentation. CGSDM aims to obtain the information of different semantic categories through K-Means clustering algorithm. CAFEM performs an average operation on the feature vectors in each semantic category to obtain semantic consistency information. RAFEM deeply excavates the information contained in each semantic category through the modeling method with self attention mechanism as the core, making the relationship between pixels within each semantic category to be better understood by the model. ODMSCB classifies the feature map according to the balance of different semantic categories, so that different decoders can be applied to feature maps with different semantic category balance. These four parts complement each other, greatly improving the model’s anti-interference ability while also enhancing the ability to handle category imbalance issue. We compared our method with several of the most advanced deep learning methods on the Vaihingen and Potsdam data sets. The final results demonstrate the superiority of our method.

Long-term chlorophyll-a (Chl-a) prediction has the potential to provide an early warning of red tide, and support fishery management and marine ecosystem health. The existing learning-based Chl-a prediction methods mostly predict a single point or multiple points with monitoring data. However, the monitoring data are subject to sparse sampling and difficult to be measured in a large-scale and synchronous way. Moreover, the advanced learning-based models for point Chl-a prediction, such as long short-term memory (LSTM) and convolutional neural network (CNN)-LSTM, are unable to fully mining the spatio-temporal correlation of Chl-a variations. Therefore, by using the satellite remote sensing data with extensive coverage, we design a framework, namely Ca-STANet, to simultaneously predict the Chl-a of all the locations in a large-scale area from the perspective of spatio-temporal field. Specifically in our method, the original data are firstly divided into multiple sub-regions to capture the spatial heterogeneity of large-scale area. Then, two modules are respectively operated to mine the spatial correlation and long-term dependency features. Finally, the outputs from the two modules are integrated by a fusion module to fully mine the spatio-temporal correlations, which are exploited to attain the final Chl-a prediction. In this paper, the proposed Ca-STANet is comprehensively evaluated and compared with the legacy methods based on the OC-CCI Chl-a 5.0 data of the Bohai Sea. The results demonstrate that the proposed Ca-STANet is highly effective for Chl-a prediction and achieves higher prediction accuracy than the baseline methods. Moreover, as the OC-CCI Chl-a 5.0 data have many missing areas, we introduce DINEOF method to fill the data gaps before using them for prediction.

Changes in oceanic variables, such as sea surface temperature (SST) and chlorophyll-a (Chl-a), have important implications for marine ecosystems and global climate change. The deep learning methods relying on convolutional neural networks can be employed to extract the spatial correlation for the prediction of oceanic variables. However, these methods are inflexible in the cases where some regions, e.g., land and islands, are invalid for the prediction of oceanic variables. By contrast, the graph convolutional network (GCN) is capable of capturing the large-scale spatial dependency existing in the irregular data. Owing to this, in this paper, we propose a GCN-based method for the prediction of oceanic variables, including SST and Chl-a, with high accuracy, which is referred to as OVPGCN. The proposed OVPGCN consists of three modules aiming to fully extract the spatial correlation and temporal dependency via modeling the multi-characteristics of the spatio-temporal dynamic evolution. In particular, three modules are implemented to extract the stationary and non-stationary variations in the recent spatio-temporal sequences, the spatial differences between different sites, and the periodic features in historical data, respectively. The well-designed OVPGCN is applied to the monthly SST and Chl-a prediction in the Bohai Sea and the Northern South China Sea (NSCS). The performance demonstrates that the proposed OVPGCN is highly effective and enables to achieve much higher prediction accuracy than the state-of-the-art methods.

Low-order features based on convolution kernel are easy to be distorted when encountering dramatic view angle transformation and atmospheric scattering in remote sensing (RS) images. To address this concern, this article first proposes to operate semantic segmentation of RS images based on the high-order information, which can represent the relative relationship of low-order features and is robust and stable when suffering feature distortion. Besides, semantic decouples have recently been well researched and have achieved significant improvement in image understanding. Thus, in this article, a high-order semantic decoupling network (HSDN) is proposed to disentangle features by semantics based on high-order features. Specifically, HSDN first represents each pixel by calculating the pixel-level affinity as a high-order feature and then clusters these pixels into different semantics. Afterward, an attention-like mask generation module is designed for both intra-semantic and inter-semantic groups, leading to three kinds of masks, including the semantic decoupling mask (SDM), which utilizes each high-order cluster centroid as a mask to compact features intracluster and expand different interclusters, so as to improve semantic disentangle performance to a better extent; semantic enhancement mask (SEM), which records pixel-level relative correlation within a class to sufficiently exploit high-order features and could enhance feature robustness; and boundary supplementary mask (BSM), which aims to process borderline pixels to reduce cluster errors. Finally, by applying masks on pixels both within classes and on borderlines, semantic decoupled features are generated and concatenated to realize segmentation. The quantitative and qualitative experiments are conducted on two large-scale fine-resolution RS image datasets to demonstrate the significant performance of adopting high-order representation. Besides, we also implement numerous experiments to validate the effectiveness of the proposed semantic decouple framework in dealing with complicated and distortion-prone RS image segmentation tasks.

With the development of computer vision, the semantic segmentation of remote sensing images, which has become an important topic, has been utilized in various applications for image content analysis and understanding, such as urban planning, natural disaster monitoring, and land resource management. Many approaches have been proposed to address these problems. However, due to obvious differences in resolution, spatial structure, and semantics between remote sensing images and ordinary images, the semantic segmentation of remote sensing images is still challenging. In this paper, we propose a novel multiscale image generation network (MIGN) that can efficiently generate high-resolution segmentation results by considering both details and boundary information. In particular, a multi-attention mechanism method for semantic segmentation of remote sensing images is designed. The attention weight is calculated by capturing the interaction of cross dimensions in a two-branch structure, which can learn the underlying feature information and guarantee the performance of each pixel feature for final classification. We also propose an edge supervised module to ensure that the segmentation boundary has a more accurate performance. A multiscale image fusion algorithm based on the Bayes model is proposed to improve the accuracy of the segmentation module. The performance of our model is evaluated on the ISPRS Vaihingen and Potsdam datasets. The results show that our method is superior to the most advanced image segmentation methods in terms of MIoU and pixel accuracy.

Recently, recommendation systems have been widely usedin online business scenarios, which can improve the online experience by learning the user or item characteristics to predict the user’s future behavior and to realize precision marketing. However, data sparsity and cold-start problems limit the performance of recommendation systems in some emerging fields. Thus, cross-domain recommendation has been proposed to handle the abovementioned problems. Nonetheless, many cross-domain recommendations only consider modeling a single user’s representation and ignore user-group information (this group has similar behavior and interests). Additionally, most studies are based on matrix factorization for generating embeddings, which results in a weak generalization ability of user latent features. In this paper, we propose a novel cross-domain recommendation model viaUser-Clustering andMultidimensional informationFusion (UCMF) that attempts to enhance user representation learning in a data sparsity scenario for accurate recommendation. In addition, we consider a user’s individual information and cross-domain feature information. A novel multidimensional information fusion is proposed to guarantee the robustness of the user features. In particular, we apply a graph neural network to learn the user-group features, which can effectively save the correlation among users’ information and guarantee feature performance. In other words, the Wasserstein autoencoder is utilized to learn the cross-domain user features, which can guarantee the consistency of user features from different domains. Experiments conducted on real-world datasets empirically demonstrate that our proposed method outperforms the state-of-the-art methods in cross-domain recommendation.

Semantic segmentation of remote sensing (RS) images plays a vital role in a variety of fields, including urban planning, natural disaster monitoring, and land resource management. Due to the complexity and low resolution of RS images, many approaches have been proposed to handle the related task. However, these previously developed approaches dedicate to contextual interaction but ignore the cross-scale semantic correlation and multi-scale boundary information. Therefore, we propose a Cross-scale Graph Interaction Network (CGIN) to address semantic segmentation problems of RS images, which consists of a semantic branch and a boundary branch. In the semantic branch, we first apply atrous convolution to extract multi-scale semantic features of RS images. Particularly, based on the multi-scale semantic features, a Cross-scale Graph Interaction (CGI) module is introduced, which establishes cross-scale graph structures and performs adaptive graph reasoning to capture the cross-scale semantic correlation of RS objects. In the boundary branch, we propose a Multi-scale Boundary Feature Extraction (MBFE) module that utilizes atrous convolutions with different dilation rates to extract multi-scale boundary features. Finally, to address the problem of sparse boundary pixels in the fusion process of the two branches, we propose a Multi-scale Similarity-guided Aggregation (MSA) module by calculating the similarity of semantic features and boundary features at the corresponding scale, which can emphasize the boundary information in semantic features. Our proposed CGIN outperforms state-of-the-art approaches in numerical experiments conducted on two benchmark remote sensing datasets.

Myriad of machine learning (ML) algorithms refine model parameters iteratively. Existing synchronous data-parallel frameworks can accelerate training with convergence guarantees. However, the pre-assigned workload-based synchronous design still poses great challenges, since fast workers must wait for slow, straggling ones, especially in a heterogeneous computing cluster. Asynchronous alternatives can bypass this performance bottleneck, but at expense of potentially losing convergence guarantees. This article proposes a new time-based flexible synchronous parallel framework (FSP). It provides strict convergence analysis by consistently updating parameters, as well as significant cost reduction by completely unleashing the power of fast workers. It identifies the optimal synchronization frequency, by online balancing costs of parameters’ update and benefits brought by their freshness. Besides the basic goal of keeping all workers fully CPU-utilized, FSP also aims to keep data spread over the cluster fully utilized, so that they can contribute to convergence with equal opportunities. These proposals are all implemented in a prototype system Flegel, with additional engineering optimizations for further performance enhancement and programming facilitation. Experiments demonstrate that Flegel significantly outperforms recent studies.

Cross-domain recommendation is an effective way to cope with the cold-start problem in recommendation systems. Knowledge of the current, particularly reviews, is taken into account to improve user/item embedding to reduce the neg-ative transfer that occurs during mapping processes across the source and target domains. Traditional approaches, on the other hand, typically apply review information from the source and target domain independently without consideration of user preference divergence. In this paper, we propose a novel Review-based Preference Invariance Transfer Network (RPITN) to minimize negative transfer by combining reviews from two domains. We first build a review preference invari-ance (RPI) embedding procedure to express user/item review correlations between two domains. Then, to improve the gen-eralization ability of user/item embedding and prevent negative transfer across domains, we carefully insert RPI into the embedding learning and mapping process. Extensive exper-iments on real-world datasets demonstrate the superiority of RPITN compared with other recommendation methods.

Image manipulation localization is of vital importance to public order protection. One dominant approach is to detect the anomalies in images, i.e., visual artifacts, as the tampered edge clue for aiding manipulation prediction. Nevertheless, we argue that these methods struggle with the modeling of spatial inconsistency within multi-scale, resulting in sub-optimal model performance. To overcome this problem, in this paper, we propose a novel end-to-end method to identify the multi-scale spatial inconsistency for image manipulation localization (abbreviated as MSI) where the multi-scale edge-guided attention stream (MEA) and multi-scale context-aware search stream (MCS) are jointly explored in a unified framework, moreover, multi-scale information is efficiently used. In the former, the edge-attention module is designed to precisely locate the tampered regions based upon multi-scale edge boundary features. In the latter, the context-aware search module is designed to model spatial contextual information within multiple scales. To validate the effectiveness of the proposed method, we conduct extensive experiments on six image manipulation localization datasets including NIST-2016, Columbia, CASIA1.0, COVER, DEF-12K, and IMD2020. The experimental results demonstrate that our proposed method can outperform state-of-the-art methods by a significant margin in terms of average F1 score while maintaining robustness with respect to various attacks. Compared with MVSS-Net (Published in ICCV 2021) on the NIST-2016, CASIA1.0, DEF-12K, and IMD2020 datasets, the improvements in F1 score can reach 6.7%, 9.5%, 5.4%, and 8.4%, respectively.

With the development of deep neural networks, especially generation networks, gray image coloring technology has made great progress. As one of the fields, remote sensing image colorization needs to be solved urgently. This is because remote sensing images cannot obtain clear color images due to the limitations of shooting equipment and transmission equipment. Compared with ordinary images, remote sensing images are characterized by the uneven spatial distribution of objects, therefore, it is a great challenge to ensure the spatial consistency of coloring. To embrace this challenge, we propose a new joint stream DCGAN including a micro stream and a macro stream, in which the latter is set as a prior to constrain the former for colorization. In addition, the Low-level Correlation Feature Extraction (LCFE) module is proposed to obtain the salient shallow detail feature with global correlation, which is used to enhance the global constraints as well as supplement the low-level information to the micro stream. What's more, we propose the Gated Selection (GSM) module by selecting useful information using a gated scheme to fuse features from two streams appropriately. Comprehensive comparison and ablation experiments are implemented and verify the proposed method performs surpasses other methods in both qualitative and quantitative metrics.

Retrieving 3D shapes based on 2D images is a challenging research topic, due to the significant gap between different domains. Recently, various approaches have been proposed to handle this problem. However, the majority of methods target the cross-domain retrieval task as a pure domain adaptation problem, which focuses on the alignment but ignores the visual relevance between the 2D images and their corresponding 3D shapes. To fundamentally decrease the divergence between different domains, we propose a novel cross-domain learning network (CLN) for 2D image-based 3D shape retrieval task. First, we estimate the pose information from the 2D image to guide the view rendering of 3D shapes, which increases the visual correlations of the cross-domain data to eliminate the divergence between them. Second, we introduce a novel joint learning network, considering both the domain-specific characteristics and the cross-domain interactions for data alignment, which further compensates for the gap between different domains by controlling the distance of intra- and inter-classes. After the metric learning process, discriminative descriptors of images and shapes are generated for the cross-domain retrieval task. To prove the effectiveness and robustness of the proposed method, we conduct extensive experiments on the MI3DOR, SHREC’13, and SHREC’14 datasets. The experimental results demonstrate the superiority of our proposed method, and significant improvements have been achieved compared with state-of-the-art methods.

As we all know, remote sensing (RS) images contain multi-scale and numerous RS objects, along with massive and complex spatial topological relationships, such as the adjacency, proximity relations of same-scale objects, and inclusion relations of cross-scale objects. However, the existing semantic segmentation methods have never explored the cross-scale relations, which are especially important when comes to the situation that the RS objects cannot be accurately identified, they could be supplemented by the surrounding contents. To address the above concern, we propose a scale-relation joint decoupling network (SRJDN) for the semantic segmentation of RS images by simultaneously considering decoupling scales and decoupling relations to excavate more complete relationships of multi-scale RS objects. The SRJDN is performed by following three steps, namely scale decoupling (SD), relation decoupling (RD), and fine-granularity guided fusion (FGF). The SD module uses dilated convolution with different rates to decouple RS objects into different scale feature groups, from small to large scales. Afterward, the RD considers all the spatial topological relationships and decouples these relationships according to the scale, which is divided into two parts, including same-scale relation extraction (SSRE) and cross-scale relation extraction (CSRE). The SSRE establishes the graph structures at each scale independently to mine the relationships of same-scale RS objects and the CSRE constructs the graph in a unified pattern between cross-scales to explore cross-scale target relationships. Third, the FGF module regards small-scale features as fine-granularity representation and applies its attention map to guide the learning of other scale features, which could mine more reliable and comprehensive saliency information and improve the feature consistency. Numerical experiments conducted on two large-scale fine-resolution RS image datasets empirically demonstrate the robustness of the proposed joint decoupling strategy and the effectiveness of the fine-granularity guided fusion in RS image semantic segmentation tasks.

Temporal action localization is currently an active research topic in computer vision and machine learning due to its usage in smart surveillance. It is a challenging problem since the categories of the actions must be classified in untrimmed videos and the start and end of the actions need to be accurately found. Although many temporal action localization methods have been proposed, they require substantial amounts of computational resources for the training and inference processes. To solve these issues, in this work, a novel temporal-aware relation and attention network (abbreviated as TRA) is proposed for the temporal action localization task. TRA has an anchor-free and end-to-end architecture that fully uses temporal-aware information. Specifically, a temporal self-attention module is first designed to determine the relationship between different temporal positions, and more weight is given to features within the actions. Then, a multiple temporal aggregation module is constructed to aggregate the temporal domain information. Finally, a graph relation module is designed to obtain the aggregated graph features, which are used to refine the boundaries and classification results. Most importantly, these three modules are jointly explored in a unified framework, and temporal awareness is always fully used. Extensive experiments demonstrate that the proposed method can outperform all state-of-the-art methods on the THUMOS14 dataset with an average mAP that reaches 67.6% and obtain a comparable result on the ActivityNet1.3 dataset with an average mAP that reaches 34.4%. Compared with A2Net (TIP20), PCG-TAL (TIP21), and AFSD (CVPR21) TRA can achieve improvements of 11.7%, 4.4%, and 1.8%, respectively on the THUMOS14 dataset.

The accuracy of the semantic segmentation results of ships is of great significance to coastline navigation, resource management, and territorial protection. Although the ship semantic segmentation method based on deep learning has made great progress, there is still the problem of not exploring the correlation between the targets. In order to avoid the above problems, this paper designed a multi-scale graph convolutional network and dynamic iterative class loss for ship segmentation in remote sensing images to generate more accurate segmentation results. Based on DeepLabv3+, our network uses deep convolutional networks and atrous convolutions for multi-scale feature extraction. In particular, for multi-scale semantic features, we propose to construct a Multi-Scale Graph Convolution Network (MSGCN) to introduce semantic correlation information for pixel feature learning by GCN, which enhances the segmentation result of ship objects. In addition, we propose a Dynamic Iterative Class Loss (DICL) based on iterative batch-wise class rectification instead of pre-computing the fixed weights over the whole dataset, which solves the problem of imbalance between positive and negative samples. We compared the proposed algorithm with the most advanced deep learning target detection methods and ship detection methods and proved the superiority of our method. On a High-Resolution SAR Images Dataset [1], ship detection and instance segmentation can be implemented well.

The semantic segmentation of frazil ice and anchor ice is of great significance for river management, ship navigation, and ice hazard forecasting in cold regions. Especially, distinguishing frazil ice from sediment-carrying anchor ice can increase the estimation accuracy of the sediment transportation capacity of the river. Although the river ice semantic segmentation methods based on deep learning has achieved great prediction accuracy, there is still the problem of insufficient feature extraction. To address this problem, we proposed a Fine-Grained River Ice Semantic Segmentation (FGRIS) based on attentive features and enhancing feature fusion to deal with these challenges. First, we propose a Dual-Attention Mechanism (DAM) method, which uses a combination of channel attention features and position attention features to extract more comprehensive semantic features. Then, we proposed a novel Branch Feature Fusion (BFF) module to bridge the semantic feature gap between high-level feature semantic features and low-level semantic features, which is robust to different scales. Experimental results conducted on Alberta River Ice Segmentation Dataset demonstrate the superiority of the proposed method.

Visual relationship detection aims to describe the interactions between pairs of objects, such asperson-ride-bikeandbike-next to-cartriplets. In reality, it is often the case that there exist some groups of strongly correlated relationships, while others are weakly related. Intuitively, the common relationships can be roughly categorized into several types such as geometric (e.g., next to), action (e.g., ride), and so on. However, previous studies ignore the relatedness discovery among multiple relationships, which only lie on a unified space to leverage visual features or statistical dependencies into categories. To tackle this problem, we propose an adaptively clustering-driven network for visual relationship detection, which can implicitly divide the unified relationship space into several subspaces with specific characteristics. Particularly, we propose two novel modules to discover the common distribution space and latent relationship association, respectively, which map pairs of object features into translation subspaces to induce the discriminative relationship clustering. Then, a fused inference is designed to integrate the group-induced representations with the language prior to facilitate the predicate inference. Especially, we design the Frobenius-norm regularization to boost the clustering. To the best of our knowledge, the proposed method is the first supervised framework to realizesubject-predicate-objectrelationship-aware clustering for visual relationship detection. Extensive experiments show that the proposed method can achieve competing performances against the state-of-the-art methods on the Visual Genome dataset. Additional ablation studies further validate its effectiveness.

2D image based 3D model retrieval is a challenging research topic in the field of 3D model retrieval. The huge gap between two modalities - 2D image and 3D model, extremely constrains the retrieval performance. In order to handle this problem, we propose a novel multi-branch graph convolution network (M-GCN) to address the 2D image based 3D model retrieval problem. First, we compute the similarity between 2D image and 3D model based on visual information to construct one cross-modalities graph model, which can provide the original relationship between image and 3D model. However, this relationship is not accurate because of the difference of modalities. Thus, the multi-head attention mechanism is employed to generate a set of fully connected edge-weighted graphs, which can predict the hidden relationship between 2D image and 3D model to further strengthen the correlation for the embedding generation of nodes. Finally, we apply the max-pooling operation to fuse the multi-graphs information and generate the fusion embeddings of nodes for retrieval. To validate the performance of our method, we evaluated M-GCN on the MI3DOR dataset, Shrec 2018 track and Shrec 2014 track. The experimental results demonstrate the superiority of our proposed method over the state-of-the-art methods.

With the development of both hardware and deep neural network technologies, tremendous improvements have been achieved in the performance of automatic emotion recognition (AER) based on the video data. However, AER is still a challenging task due to subtle expression, abstract concept of emotion and the representation of multi-modal information. Most proposed approaches focus on the multi-modal feature learning and fusion strategy, which pay more attention to the characteristic of a single video and ignore the correlation among the videos. To explore this correlation, in this paper, we propose a novel correlation-based graph convolutional network (C-GCN) for AER, which can comprehensively consider the correlation of the intra-class and inter-class videos for feature learning and information fusion. More specifically, we introduce the graph model to represent the correlation among the videos. This correlated information can help to improve the discrimination of node features in the progress of graph convolutional network. Meanwhile, the multi-head attention mechanism is applied to predict the hidden relationship among the videos, which can strengthen the inter-class correlation to improve the performance of classifiers. The C-GCN is evaluated on the AFEW datasets and eNTERFACE 05 dataset. The final experimental results demonstrate the superiority of our proposed method over the state-of-the-art methods.

2D image-based 3D object retrieval provides a convenient way to manage 3D big data with easily accessed 2D images. It is also a challenging task due to the significant differences between 2D images and 3D objects. In this paper, we propose a 2D image-based 3D object retrieval method, which can reduce the distribution discrepancy between 2D images and 3D objects and learn invariant features between them. Specifically, we first construct an intermediate domain module based on maximum mean discrepancy (MMD) in an unsupervised way, which can reduce the 2D and 3D distribution discrepancy by marginal distribution constraint. Second, to further reduce conditional distribution discrepancy and learn invariant features, we use source domain labels as semantic information to dynamically guide distribution alignment. Moreover, in order to support the research in 3D object retrieval, we contribute a new dataset, MDI3D. We conducted extensive experiments on MDI3D and some popular datasets, such as MI3DOR and SHREC2013. The experimental results demonstrate the superiority of the proposed method by comparing with the state-of-the-art methods.

Three-dimensional (3D) shape recognition is a popular topic and has potential application value in the field of computer vision. With the recent proliferation of deep learning, various deep learning models have achieved state-of-the-art performance. Among them, multiview-based 3D shape representation has received increased attention in recent years, and related approaches have shown significant improvement in 3D shape recognition. However, these methods focus on feature learning based on the design of the network and ignore the correlation among views. In this article, we propose a novel progressive feature guide learning network (PGNet) that focuses on the correlation among multiple views and integrates multiple modalities for 3D shape recognition. In particular, we propose two information fusion schemes from visual and feature aspects. The visual fusion scheme focuses on the view level and employs the soft-attention model to define the weights of views for visual information fusion. The feature fusion scheme focuses on the feature dimension information and employs the quantified feature as the mask to further optimize the feature. These two schemes jointly construct a PGNet for 3D shape representation. The classic ModelNet40 and ShapeNetCore55 datasets are applied to demonstrate the performance of our approach. The corresponding experiment also demonstrates the superiority of our approach.

In this work, we aim to investigate the practical task of flexible fashion search with attribute manipulation, where users can retrieve the target fashion items by replacing the unwanted attributes of an available query image with the desired ones (e.g., changing the collar attribute from v-neck to round). Although several pioneer efforts have been dedicated to fulfilling the task, they mainly ignore the potential of generative models in enhancing the visual understanding of target fashion items. To this end, we propose an end-to-end generative attribute manipulation scheme, which consists of a generator and a discriminator. The generator works on producing the prototype image that meets the user's requirement of attribute manipulation over the query image with the regularization of visual-semantic consistency and pixel-wise consistency. Besides, the discriminator aims to jointly fulfill the semantic learning towards correct attribute manipulation and adversarial metric learning for fashion search. Pertaining to the adversarial metric learning, we provide two general paradigms: the pair-based scheme and the triplet-based scheme, where the fake generated prototype images that closely resemble the ground truth images of target items are incorporated as hard negative samples to boost the model performance. Extensive experiments on two real-world datasets verify the effectiveness of our scheme.

Image captioning is one of the most challenging tasks in AI because it requires an understanding of both complex visuals and natural language. Because image captioning is essentially a sequential prediction task, recent advances in image captioning have used reinforcement learning (RL) to better explore the dynamics of word-by-word generation. However, the existing RL-based image captioning methods rely primarily on a single policy network and reward function-an approach that is not well matched to the multi-level (word and sentence) and multi-modal (vision and language) nature of the task. To solve this problem, we propose a novel multi-level policy and reward RL framework for image captioning that can be easily integrated with RNN-based captioning models, language metrics, or visual-semantic functions for optimization. Specifically, the proposed framework includes two modules: 1) a multi-level policy network that jointly updates the word- and sentence-level policies for word generation; and 2) a multi-level reward function that collaboratively leverages both a vision-language reward and a language-language reward to guide the policy. Furthermore, we propose a guidance term to bridge the policy and the reward for RL optimization. The extensive experiments on the MSCOCO and Flickr30k datasets and the analyses show that the proposed framework achieves competitive performances on a variety of evaluation metrics. In addition, we conduct ablation studies on multiple variants of the proposed framework and explore several representative image captioning models and metrics for the word-level policy network and the language-language reward function to evaluate the generalization ability of the proposed framework.

In this paper, we propose the multi-view saliency guided deep neural network (MVSG-DNN) for 3D object retrieval and classification. This method mainly consists of three key modules. First, the module of model projection rendering is employed to capture the multiple views of one 3D object. Second, the module of visual context learning applies the basic Convolutional Neural Networks for visual feature extraction of individual views and then employs the saliency LSTM to adaptively select the representative views based on multi-view context. Finally, with these information, the module of multi-view representation learning can generate the compile 3D object descriptors with the designed classification LSTM for 3D object retrieval and classification. The proposed MVSG-DNN has two main contributions: 1) It can jointly realize the selection of representative views and the similarity measure by fully exploiting multi-view context; 2) It can discover the discriminative structure of multi-view sequence without constraints of specific camera settings. Consequently, it can support flexible 3D object retrieval and classification for real applications by avoiding the required camera settings. Extensive comparison experiments on ModelNet10, ModelNet40, and ShapeNetCore55 demonstrate the superiority of MVSG-DNN against the state-of-art methods.

In this article, we propose a novel method to address the two-dimensional (2D) image-based 3D object retrieval problem. First, we extract a set of virtual views to represent each 3D object. Then, a soft-attention model is utilized to find the weight of each view to select one characteristic view for each 3D object. Second, we propose a novel Holistic Generative Adversarial Network (HGAN) to solve the cross-domain feature representation problem and make the feature space of virtual characteristic view more inclined to the feature space of the real picture. This will effectively mitigate the distribution discrepancies across the 2D image domains and 3D object domains. Finally, we utilize the generative model of the HGAN to obtain the “virtual real image” of each 3D object and make the characteristic view of the 3D object and real picture possess the same feature space for retrieval. To demonstrate the performance of our approach, We established a new dataset that includes pairs of 2D images and 3D objects, where the 3D objects are based on the ModelNet40 dataset. The experimental results demonstrate the superiority of our proposed method over the state-of-the-art methods.

Due to the advantages of retrieval speed and storage space, deep hashing methods have become a research hotspot in the field of large-scale image retrieval. Most of existing deep hashing methods pay close attention to similarity between images without images at the top of the ranking list similar to query targets. In the paper, a novel deep hashing model is proposed to preserve top images similar to the query images and optimize the quality of hash codes for image retrieval. Specifically, the optimized AlexNet is utilized to extract discriminative image representations and learn hashing functions simultaneously. The loss function based on acceleration strategy is designed to ensure similarity between returned images at the top of the ranking list and query images. In addition, we implement the model training in a batch-process fashion to low the image storage. Moreover, our extensive experiments on standard benchmarks demonstrate that our method outperforms several state-of-the-art deep hashing methods.

Tracking live fish in an open underwater environment to investigate their behavior is of great value for many applications, e.g. biological and robotic research. However, tracking fish in real world environment is a challenging task due to complex non-rigid deformation and abrupt movement of fish. In this paper, we explore and incorporate motion property of fish and propose a real-time fish tracking method based on novel adaptive multi-appearance models and tracking strategy, which can be adapted to various changes of the fish appearance caused by non-rigid deformation. Experimental results show the promising performance of the proposed method can outperform the previous method by 13.4% in accuracy on average and is robust to real-time underwater fish tracking.

Exercise is considered as an effective mean against overweight and obesity-related diseases. In this paper, a real-time activity recognition and counting approach is proposed to evaluate amount of exercise only using a wearable smart watch. First, accelerometer and gyroscope data are collected to extract efficient features. Then Support Vector Machine classifiers are trained to recognize nine common exercise activities in real time. In order to measure the frequency of repetitive activity, a general activity counting algorithm based on gyroscope is proposed which is applicable for different types of activity. Various activities can be counted uninterruptedly using the proposed general method without frequently changing algorithms. Through experiments, it is demonstrated that the extracted features are efficient for real time exercise activity recognition. Moreover, our comparative experiments have shown that our counting approach is more accurate than other products on the market.

Whenever looking at a stranger's portrait, besides observable appearance, we always build a personality impression implicitly in our subconscious. It is quite interesting to ask how a portrait impresses people. This paper presents a novel method to infer personality impression from portrait. Firstly, a questionnaire is applied to demonstrate the consistence of people's impression. And then personality-related features are explored through the statistical analysis method. Finally, features are trained using Support Vector Machine. Experimental results demonstrate our method could achieve a precision of 52.14% and a recall of 52.78% on inferring 4 personalities from 2,463 randomly selected portraits of people downloaded from "Google images". Improvements of 44.04% and 37.91% are reported compared to a baseline method. And features contribution analysis deeply unveils the correspondence between portrait contents and personality impressions. Demonstrations with respect to visual patterns in portrait collages of different personalities further prove the effectiveness of the proposed method. Furthermore, we apply our method to analyze portraits of Hillary Clinton and obtain an interesting multifaceted figure of this famous politics, which is another proof of both our concept and method.

A new color image watermarking algorithm with resistance to JPEG lossy compression based on quantization index modulation (QIM) is proposed in this paper. As it is known, QIM method can achieve a good balance between the embedding bit rate, robustness and distortion between the original image and the composited image by modulating the source signal into different clusters. The corresponding DCT coefficients margins of any two color channels selected from the three as the source signal is substantiated could achieve a high level embedding robustness and a low level distortion in this paper. However, JPEG lossy compression could bring a destructive influence to the watermark since it discards pretty much image information. In this paper, a compensation function is designed to correct the errors caused by JPEG compression. Experiments show the algorithm proposed has a good performance to resist the high compression ratio JPEG lossy compression and other attacks.