Deep learning exhibits powerful capability in image inpainting task, particularly in generating pixel-level details closely with the human visual perception. However, the complex background or larger missing regions make it still encounters the artifacts. Many researchers have investigated that prior information is crucial for guiding the image inpainting. In this article, we introduce the dual-attention mechanism, including lightweight spatial attention and linearized attention, to construct an end-to-end texture and structure-guided image inpainting method. In the first stage, we build the detail inpainting network with the lightweight spatial attention. In this model, the extracted texture and structural features are fused with multi-layers and then the fused detail image is considered as the prior to guide the detail repair of corrupted images. In the second stage, we construct the content completing network by the repaired detail and the linearized Transformer module. This module not only overcomes the limitation of the receptive field size of convolutional kernels that can improve the long-range modeling of features but also can significantly reduce the computational complexity of the original Transformer. To demonstrate the superior effectiveness of the proposed method, we perform extensive experiments with advanced models on three datasets: CelebA-HQ, Places2, and Paris Street Views. Comparative results manifest that our method achieves excellent image inpainting results that are conform to the human visual system. The code is available at https://github.com/QinLab-WFU/TSGDAM

Deep hashing has attracted broad interest in cross-modal retrieval because of its low cost and efficient retrieval benefits. To capture the semantic information of raw samples and alleviate the semantic gap, supervised cross-modal hashing methods that utilize label information which could map raw samples from different modalities into a unified common space, are proposed. Although making great progress, existing deep cross-modal hashing methods are suffering from some problems, such as: 1) considering multi-label cross-modal retrieval, proxy-based methods ignore the data-to-data relations and fail to explore the combination of the different categories profoundly, which could lead to some samples without common categories being embedded in the vicinity; 2) for feature representation, image feature extractors containing multiple convolutional layers cannot fully obtain global information of images, which results in the generation of sub-optimal binary hash codes. In this paper, by extending the proxy-based mechanism to multi-label cross-modal retrieval, we propose a novel Deep Semantic-aware Proxy Hashing (DSPH) framework, which could embed multi-modal multi-label data into a uniform discrete space and capture fine-grained semantic relations between raw samples. Specifically, by learning multi-modal multi-label proxy terms and multi-modal irrelevant terms jointly, the semantic-aware proxy loss is designed to capture multi-label correlations and preserve the correct fine-grained similarity ranking among samples, alleviating inter-modal semantic gaps. In addition, for feature representation, two transformer encoders are proposed as backbone networks for images and text, respectively, in which the image transformer encoder is introduced to obtain global information of the input image by modeling long-range visual dependencies. We have conducted extensive experiments on three baseline multi-label datasets, and the experimental results show that our DSPH framework achieves better performance than state-of-the-art cross-modal hashing methods. The code for the implementation of our DSPH framework is available athttps://github.com/QinLab-WFU/DSPH.

Deep hashing integrates the advantages of deep learning and hashing technology, and has become the mainstream of the large-scale image retrieval field. However, when training the deep hashing models, most of the existing approaches regard the similarity margin of image pairs as a constant. Once similarity distance exceeds the fixed margin, the network will not learn anything, which easily results in model collapses. In this paper, we address this dilemma with a novel unified deep hashing framework, termed Deep Neighborhood Structure-preserving Hashing (DNSH), to generate the similarity-preserving and discriminative hash codes. Specifically, by extracting the discriminative object characteristics with large variances, we design an adaptive margin quadruplet loss to further explore the underlying similarity relationship between image pairs, reflecting the correct semantic structure among its neighbors. Based on the quadruple form, we develop a quadruple regularization to decrease quantization errors between binary-like embedding and hashing codes. Furthermore, through learning bit balance and bit independent terms jointly, we present the binary code constraint loss to alleviate redundancy in different bits. Extensive evaluations on four popular benchmark datasets demonstrate that our proposed deep hashing framework achieves an excellent performance than the comparison methods.

Cross-modal retrieval methods based on hashing have gained significant attention in both academic and industrial research. Deep learning techniques have played a crucial role in advancing supervised cross-modal hashing methods, leading to significant practical improvements. Despite these achievements, current deep cross-modal hashing still encounters some underexplored limitations. Specifically, most of the available deep hashing usually utilizes pair-wise or triplet-wise strategies to promote the separation of the inter-classes by calculating the relative similarities between samples, weakening the compactness of intra-class data from different modalities, which could generate ambiguous neighborhoods. In this article, the Deep Neighborhood-aware Proxy Hashing (DNPH) framework is proposed to learn a discriminative embedding space with the original neighborhood relation preserved. By introducing learnable shared category proxies, the neighborhood-aware proxy loss is proposed to project the heterogeneous data into a unified common embedding, in which the sample is pulled closer to the corresponding category proxy and is pushed away from other proxies, capturing small within-class scatter and big between-class scatter. To enhance the quality of the obtained binary codes, the uniform distribution constraint is developed to make each hash bit independently obey the discrete uniform distribution. In addition, the discrimination loss is designed to preserve modality-specific semantic information of samples. Extensive experiments are performed on three benchmark datasets to prove that our proposed DNPH framework achieves comparable or even better performance compared with the state-of-the-art cross-modal retrieval applications. The corresponding code implementation of our DNPH framework is as follows: https://github.com/QinLab-WFU/OUR-DNPH .

With the preferable efficiency in storage and computation, hashing has shown potential application in large-scale multimedia retrieval. Compared with traditional hashing algorithms using hand-crafted characteristics, deep hashing inherits the representational capacity of deep neural networks to jointly learn semantic features and hash functions, encoding raw data into compact binary codes with significant discrimination. Generally, most of the current multi-wise hashing methods view the similarity margins between image pairs as constant values in training process. When the distance between sample pairs exceeds the fixed margin, the hashing network would not learn anything. Besides, available hashing methods commonly introduce the random sampling strategy to build training batches and ignore the sample distribution, which is harmful to parameter optimization. In this paper, we propose a novel Deep Adaptive Quadruplet Hashing with probability sampling (DAQH) for discriminative binary code learning. Specifically, with exploring the distribution relationship of raw samples, a non-uniform probability sampling strategy is proposed to build more informative and representative training batches, while maintaining the diversity of training samples. By introducing the prior similarity of sample pairs to calculate corresponding margins, an adaptive margin quadruplet loss is designed to dynamically preserve the underlying semantic relationships with its neighbors. To tune the attributes of binary codes, by combining quadruple regularization and orthogonality optimization, binary code constraint is developed to make the learned embedding with significant discrimination. Extensive experimental results on various benchmark datasets demonstrate our proposed DAQH framework achieves state-of-the-art visual similarity search performance.

As a classical clustering model, Gaussian Mixture Model (GMM) can be the footstone of dominant machine learning methods like transfer learning. Evolving GMM is an approximation to the classical GMM under time-critical or memory-critical application scenarios. Such applications often have constraints on time-to-answer or high data volume, and raise high computation demand. A prominent approach to address the demand is GPGPU-powered computing. However, the existing evolving GMM algorithms are confronted with a dilemma between clustering accuracy and parallelism. Point-wise algorithms achieve high accuracy but exhibit limited parallelism due to point-evolutionary pattern. Block-wise algorithms tend to exhibit higher parallelism. Whereas, it is challenging to achieve high accuracy under a block-evolutionary pattern due to the fact that it is difficult to track evolving process of the mixture model in fine granularity. Consequently, the existing block-wise algorithm suffers from significant accuracy degradation, compared to its batch-mode counterpart: the standard EM algorithm. To cope with this dilemma, we focus on the accuracy issue and develop an improved block-evolutionary GMM algorithm for GPGPU-powered computing systems. Our algorithm leverages evolving history of the model to estimate the latest model order in each incremental clustering step. With this model order as a constraint, we can perform similarity test in an elastic manner. Finally, we analyze the evolving history of both mixture components and the data points, and propose our method to merge similar components. Experiments on real images show that our algorithm significantly improves accuracy of the original general purpose bock-wise algorithm. The accuracy of our algorithm is at least comparable to that of the standard EM algorithm and even outperforms the latter under certain scenarios.