Cross-modal retrieval tasks, encompassing the retrieval of image–text, video–audio, and more, are progressively gaining significance in response to the exponential growth of information on the Internet. However, there has always been a cloud hanging over multimodal tasks due to the inherent challenges in aligning different modalities with distinct physical meanings. Most previous works simply rely on a single multimodal encoder or a novel similarity calculation for fusion, which often result in unsatisfactory performance. To tackle this challenge, we introduce a Momentum Hypergraph Convolutional Network (MoHGCN) for multimodal representation learning, which strengthens the alignment of both visual and textual data before the retrieval process. Specifically, MoHGCN utilizes contrastive learning to select the most challenging negative and positive samples to form hyperedges and completes the modality alignment through two rounds of fusion. Subsequently, the fully integrated node features and global features are fused using a fusion encoder to obtain the final multimodal representation vector for image–text retrieval. Extensive experiments are conducted on two widely used datasets, namely Flickr30K and MSCOCO, to demonstrate the superiority of the proposed MoHGCN approach in achieving the state-of-the-art performances.

Pattern images are artificially designed images that possess distinctiveness in their elements, styles, and arrangements. With the ever-growing number of pattern images, pattern image retrieval emerges as a promising technique with significant potential for commercial and industrial applications, such as fashion and home decoration, facilitating rapid identification of preferred print patterns by users. The main purpose of multi-label pattern image retrieval is to effectively represent and match images with their corresponding labels. Compared to conventional image retrieval, multi-label pattern image retrieval faces greater challenges due to the richer semantic information contained within the abstract print patterns and the complex relationships between multiple labels. To tackle these challenges, we propose a model specifically designed for multi-label pattern image retrieval, called Tran-GCN. Our proposed model is built upon a Transformer-based autoregressive architecture, which leverages image information to guide the exploration of correlations between different labels through the textual modality. By utilizing this correlation information, we construct a graph convolutional network (GCN) model to further enhance the correlations between image and label representations. To be more specific, our Tran-GCN model utilizes a cross-modal attention mechanisms at each layer to effectively aggregate visual features from the input image and update label semantics through residual connections. The GCN module is updated based on the correlation between textual features, as represented in a relationship matrix. Extensive experiments on two widely used public visual benchmarks, MS-COCO and NUS-WIDE, as well as a multi-label pattern image dataset, Pattern 2, consistently demonstrate the ability of our proposed Tran-GCN model for general use and its superior performance in multi-label pattern image retrieval tasks as well.