Hashing learning has moved into the mainstream for multimedia retrieval because it offers the advantages of low storage cost and high retrieval efficiency. Currently, most cross-modal hashing methods commonly explore the similarity relations between samples by constructing pair-wise or triplet-wise constraints. However, these methods focus on the relative correct ranking of samples, ignore the potential semantic similarity of raw sample distribution, and generate sub-optimal hash codes. To resolve this issue, the novel Deep Potential Semantic-aware Hashing framework (DPSaH) is proposed to mine the local semantic structure of heterogeneous samples, maintaining inter-modality-consistent and cross-modality-correlated semantic relationships. Specifically, by exploring the potential local structure of the data, the multi-modal quadruple loss is extended to the cross-modal hashing framework, thereby preserving the potential semantic neighborhoods among raw samples in Hamming space. During model training, based on the average semantic labels, the label-averaged balanced strategy is developed to quantify the frequency difference between positive and negative samples. Besides, by injecting noise information into the generated discrete codes, the binary-injection loss is introduced to alleviate the over-activation of specific bits, decorrelating different bits in the Hamming space. Extensive experiments are performed on three public datasets, and the results verify the superiority of the DPSaH framework compared to the current mainstream cross-modal hashing frameworks. The source code for DPSaH is available at https://github.com/QinLab-WFU/DPSaH .

Unsupervised learning is one of the fundamental machine learning methods. Clustering is a vital unsupervised learning task and can significantly contribute to the detection of hidden structures in unknown datasets. Clusterability is an important concept due to the fact that it can theoretically portray the extent to which a clustering algorithm can recover a benchmark clustering, with the absence of excessive experimental validations. Moreover, conventional batch-mode-clustering-oriented clusterability analysis should be extended to the incremental setting when the clustering algorithm is required to handle stream data. However, such clusterability analysis is facing two barriers. First, the incremental clustering algorithm proceeds in a step-wise manner and can merely access the newly arrived data of the current step. This extremely fragmentary view of the entire input data stream inevitably results in a biased perception of the underlying benchmark clustering. Second, incremental clustering is conventionally applied to real-time or massive-data scenarios. Such application scenarios typically require the computational power of mainstream SIMD (Single Instruction Multiple Data) hardware accelerators. However, strong data dependency inherently exists between two successive steps of an incremental clustering algorithm, which dramatically impairs data parallelism. In view of these constraints, we propose our roadmap to theoretically analyze and ensure the clusterability under an incremental setting in terms of a general clusterability metric: niceness (higher intra-cluster similarity than inter-cluster similarity). In our work, a nice-k clustering (a clustering that has k clusters and satisfies the niceness metric) is supposed to exist in the input data stream. Meanwhile, the input data stream is supposed to be divided into a series of micro-clusters, and the micro-clusters are incrementally clustered into clusters. In addition, we rely on an assumption (homogeneity assumption) that every micro-cluster merely contains homogenous data. First, we point out that a vital reason for the induction of heterogeneous clusters is the lack of representative micro-clusters. We propose Theorem 1 to iteratively identify a set of 2[Formula: see text] representative micro-clusters that can cover all k benchmark clusters. Therefore, we can trade the number of clusters for homogeneity and thus assure clusterability. Second, we demonstrate that evolution in the granularity of a micro-cluster can prompt SIMD-parallelism more than in the granularity of a single data point. Consequently, the clusterability-assured method of Theorem 1 is furthermore parallel-friendly. In all, we depict a roadmap to assure clusterability under both incremental and SIMD-friendly constraints.

Currently, how to quickly retrieve target images from large-scale remote sensing data has emerged as a critical challenge in the context of explosive growth of remote sensing data volume. To deal with this challenge, hash learning becomes an ideal choice with its low storage cost and high efficiency. In recent years, the combination of hash learning with deep neural networks such as CNNs and Transformers has resulted in numerous frameworks demonstrating excellent performance. However, in the field of remote sensing image hashing, previous studies cannot simultaneously consider the effect of noise in feature extraction and loss optimization, so that their retrieval performance is greatly reduced due to noise interference. To resolve the mentioned problem, a Deep Noise-tolerant Hashing (DNtH) framework is proposed to learn the sample complexity and noise level, and adaptively reduce the weight of noisy information. Specifically, to realize the extraction of fine-grained features from information containing irrelevant samples, the noise-aware Transformer is proposed by introducing the patch-wise attention and depth-wise convolution. To reduce the interference of noisy labels on remote sensing image retrieval, an adaptive active-passive loss framework is proposed to dynamically adjust the weights of active passive loss, which learns the weight parameters through a dynamic weighted network while combining with asymmetric strategy for effective compact representation learning. The ratio of entropy to standard deviation and the probability difference are input into the above network and trained with the feature extraction network. Extensive experiments on three publicly available datasets show that the DNtH framework can adapt to noisy environments while achieving optimal performance in remote sensing image retrieval. The source code for the implementation of our DNtH framework is available at https://github.com/QinLab-WFU/DNtH.git .

Abstract The accurate detection of moving objects is essential in various applications of artificial intelligence, particularly in the field of intelligent surveillance systems. However, the moving cast shadow detection significantly decreases the precision of moving object detection because they share similar motion characteristics. To address the issue, the authors propose an innovative approach to detect moving cast shadows by combining the hybrid feature with a broad learning system (BLS). The approach involves extracting low‐level features from the input and background images based on colour constancy and texture consistency principles that are shown to be highly effective in moving cast shadow detection. The authors then utilise the BLS to create a hybrid feature and BLS uses the extracted low‐level features as input instead of the original data. BLS is an innovative form of deep learning that can map input to feature nodes and further enhance them by enhancement nodes, resulting in more compact features for classification. Finally, the authors develop an efficient and straightforward post‐processing technique to improve the accuracy of moving object detection. To evaluate the effectiveness and generalisation ability, the authors conduct extensive experiments on public ATON‐CVRR and CDnet datasets to verify the superior performance of our method by comparing with representative approaches.

Deep learning exhibits powerful capability in image inpainting task, particularly in generating pixel-level details closely with the human visual perception. However, the complex background or larger missing regions make it still encounters the artifacts. Many researchers have investigated that prior information is crucial for guiding the image inpainting. In this article, we introduce the dual-attention mechanism, including lightweight spatial attention and linearized attention, to construct an end-to-end texture and structure-guided image inpainting method. In the first stage, we build the detail inpainting network with the lightweight spatial attention. In this model, the extracted texture and structural features are fused with multi-layers and then the fused detail image is considered as the prior to guide the detail repair of corrupted images. In the second stage, we construct the content completing network by the repaired detail and the linearized Transformer module. This module not only overcomes the limitation of the receptive field size of convolutional kernels that can improve the long-range modeling of features but also can significantly reduce the computational complexity of the original Transformer. To demonstrate the superior effectiveness of the proposed method, we perform extensive experiments with advanced models on three datasets: CelebA-HQ, Places2, and Paris Street Views. Comparative results manifest that our method achieves excellent image inpainting results that are conform to the human visual system. The code is available at https://github.com/QinLab-WFU/TSGDAM

Deep hashing has attracted broad interest in cross-modal retrieval because of its low cost and efficient retrieval benefits. To capture the semantic information of raw samples and alleviate the semantic gap, supervised cross-modal hashing methods that utilize label information which could map raw samples from different modalities into a unified common space, are proposed. Although making great progress, existing deep cross-modal hashing methods are suffering from some problems, such as: 1) considering multi-label cross-modal retrieval, proxy-based methods ignore the data-to-data relations and fail to explore the combination of the different categories profoundly, which could lead to some samples without common categories being embedded in the vicinity; 2) for feature representation, image feature extractors containing multiple convolutional layers cannot fully obtain global information of images, which results in the generation of sub-optimal binary hash codes. In this paper, by extending the proxy-based mechanism to multi-label cross-modal retrieval, we propose a novel Deep Semantic-aware Proxy Hashing (DSPH) framework, which could embed multi-modal multi-label data into a uniform discrete space and capture fine-grained semantic relations between raw samples. Specifically, by learning multi-modal multi-label proxy terms and multi-modal irrelevant terms jointly, the semantic-aware proxy loss is designed to capture multi-label correlations and preserve the correct fine-grained similarity ranking among samples, alleviating inter-modal semantic gaps. In addition, for feature representation, two transformer encoders are proposed as backbone networks for images and text, respectively, in which the image transformer encoder is introduced to obtain global information of the input image by modeling long-range visual dependencies. We have conducted extensive experiments on three baseline multi-label datasets, and the experimental results show that our DSPH framework achieves better performance than state-of-the-art cross-modal hashing methods. The code for the implementation of our DSPH framework is available athttps://github.com/QinLab-WFU/DSPH.

Driven by the high nonlinearity of deep neural networks, deep hashing has achieved the pictured great potential in cross-modal retrieval applications, significantly bridging the modality gap. Current deep cross-modal hashing usually utilizes affinity matching or local ranking to capture the local semantic relationships in the learned common space, leading to high neighborhood ambiguity. Simultaneously, most of these frameworks utilize additional regularization terms or margin thresholds to enhance the overall performance, in which searching the model's hyper-parameters under mass training data would have a substantial overhead. In this paper, with a novel extension of information-theoretic measures, a novel deep cross-modal hashing method, named Deep Neighborhood-preserving Hashing (DNpH), is designed to learn a highly separable discrete space, effectively mitigating the semantic gap across different modalities. Specifically, to minimize neighborhood ambiguity, the Quadratic Spherical Mutual Information (QSMI) is first introduced into deep cross-modal hashing to separate neighbors and non-neighbors well, while it is free of tuning parameters during model training compared with other similarity measures. To optimize quadratic mutual information loss smoothly, a square clamping method is developed to improve the stability of model optimization, avoiding converging on bad local optimum. Besides, two transformer encoders are exploited as feature extractors for multi-modal samples to learn the informative semantic representations. Finally, we compare our proposed DNpH framework with various state-of-the-art cross-modal hashing on four public datasets, and large amounts of experiment results demonstrate our contributions and show that DNpH outperforms the compared baselines on different evaluation metrics. The corresponding code is available athttps://github.com/QinLab-WFU/DNpH.

Cross-modal retrieval methods based on hashing have gained significant attention in both academic and industrial research. Deep learning techniques have played a crucial role in advancing supervised cross-modal hashing methods, leading to significant practical improvements. Despite these achievements, current deep cross-modal hashing still encounters some underexplored limitations. Specifically, most of the available deep hashing usually utilizes pair-wise or triplet-wise strategies to promote the separation of the inter-classes by calculating the relative similarities between samples, weakening the compactness of intra-class data from different modalities, which could generate ambiguous neighborhoods. In this article, the Deep Neighborhood-aware Proxy Hashing (DNPH) framework is proposed to learn a discriminative embedding space with the original neighborhood relation preserved. By introducing learnable shared category proxies, the neighborhood-aware proxy loss is proposed to project the heterogeneous data into a unified common embedding, in which the sample is pulled closer to the corresponding category proxy and is pushed away from other proxies, capturing small within-class scatter and big between-class scatter. To enhance the quality of the obtained binary codes, the uniform distribution constraint is developed to make each hash bit independently obey the discrete uniform distribution. In addition, the discrimination loss is designed to preserve modality-specific semantic information of samples. Extensive experiments are performed on three benchmark datasets to prove that our proposed DNPH framework achieves comparable or even better performance compared with the state-of-the-art cross-modal retrieval applications. The corresponding code implementation of our DNPH framework is as follows: https://github.com/QinLab-WFU/OUR-DNPH .

Recently some researches have focused on the Domain-Generalization (DG) Re-ID problem that training and testing are not in the same domain distribution. To fit the unseen complex scenes, recently deep feature matching-based methods for DG Re-ID have been developed and achieved the state-of-the-arts. However, they ignored some cases in which the accuracy of key region matching is unstable at a single scale, and the bad impact of style variations for feature representations. To address the issues, we propose a novel deep image matching model named Multi-scale Transformer-based Matching Network (MTMN) for DG Re-ID problem. MTMN matches two images with multi-scale local respondence instead of fixed representations. Specifically, the Transformer is carefully modified to formulate efficient local interactions between query and gallery images in multiple scales. Moreover, the style normalization is introduced to filter out identity-irrelated features to promote the matching results. Comprehensive experiments on several DG Re-ID tasks demonstrate the superiority of the proposed method compared with the state-of-the-arts, e.g., 5.4$\%$and 2.6$\%$gains in Rank-1 and mAP on Market-1501$\rightarrow$MSMT17(V1) task.

Feature representation learning is a key issue in artificial intelligence research. Multiview multimedia data can provide rich information, which makes feature representation become one of the current research hotspots in data analysis. Recently, a large number of multiview data feature representation methods have been proposed, among which matrix factorization shows the excellent performance. Therefore, we propose an adaptive‐weighted multiview deep basis matrix factorization (AMDBMF) method that integrates matrix factorization, deep learning, and view fusion together. Specifically, we first perform deep basis matrix factorization on data of each view. Then, all views are integrated to complete the procedure of multiview feature learning. Finally, we propose an adaptive weighting strategy to fuse the low‐dimensional features of each view so that a unified feature representation can be obtained for multiview multimedia data. We also design an iterative update algorithm to optimize the objective function and justify the convergence of the optimization algorithm through numerical experiments. We conducted clustering experiments on five multiview multimedia datasets and compare the proposed method with several excellent current methods. The experimental results demonstrate that the clustering performance of the proposed method is better than those of the other comparison methods.

As a classical clustering model, Gaussian Mixture Model (GMM) can be the footstone of dominant machine learning methods like transfer learning. Evolving GMM is an approximation to the classical GMM under time-critical or memory-critical application scenarios. Such applications often have constraints on time-to-answer or high data volume, and raise high computation demand. A prominent approach to address the demand is GPGPU-powered computing. However, the existing evolving GMM algorithms are confronted with a dilemma between clustering accuracy and parallelism. Point-wise algorithms achieve high accuracy but exhibit limited parallelism due to point-evolutionary pattern. Block-wise algorithms tend to exhibit higher parallelism. Whereas, it is challenging to achieve high accuracy under a block-evolutionary pattern due to the fact that it is difficult to track evolving process of the mixture model in fine granularity. Consequently, the existing block-wise algorithm suffers from significant accuracy degradation, compared to its batch-mode counterpart: the standard EM algorithm. To cope with this dilemma, we focus on the accuracy issue and develop an improved block-evolutionary GMM algorithm for GPGPU-powered computing systems. Our algorithm leverages evolving history of the model to estimate the latest model order in each incremental clustering step. With this model order as a constraint, we can perform similarity test in an elastic manner. Finally, we analyze the evolving history of both mixture components and the data points, and propose our method to merge similar components. Experiments on real images show that our algorithm significantly improves accuracy of the original general purpose bock-wise algorithm. The accuracy of our algorithm is at least comparable to that of the standard EM algorithm and even outperforms the latter under certain scenarios.