Cross-modal matching is one of the most fundamental and widely studied tasks in the field of data science. To have a better understanding of the complicated cross-modal correspondences, the powerful attention mechanism has been widely used recently. In this paper, we propose a novel Dual Gated Attention Fusion (DGAF) unit to save cross-modal matching from heavy attention computation. Specifically, the attention unit in the main information flow is alternated to a single-head low-dimension light-weighted attention bypass which serves as a gate to selectively cast away noise in both modality. To strengthen the interaction between modalities, an auxiliary memory unit is appended. A gated memory fusion unit is designed to fuse the memorized inter-modality information into both modality streams. Extensive experiments on two benchmark datasets show that the proposed DGAF achieves good balance between the efficiency and the effectiveness.

Recently, the powerful attention mechanism has been wildly used to learn the fine-grained cross-modal correspondences. However, the trade-off between effectiveness and efficiency sometimes bothers existing attention-mechanism-based meth-ods. To address this deficiency, we propose a novel Triple-streamed architecture with a newly designed Memory fusion Transformer Encoder (Tri-MTE) for fine-grained cross-modal retrieval. Specifically, the whole model reserves the “late fusion” strategy thus ensuring efficiency. To strengthen the inter-modality interaction and improve the effectiveness, a memory fusion stream is designed and inserted between the modality streams to remember the modality-irrelevant infor-mation. Encoding such information to the modality represen-tation would significantly enhance the cross-modal retrieval performance. Finally, a bionic memory activation constrain-t is proposed to aid the learning procedure. Extensive ex-periments on two benchmark datasets show that the proposed method achieves promising results.

Cross-modal retrieval has become a hot research topic in recent years for its theoretical and practical significance. This paper proposes a new technique for learning such deep visual-semantic embedding that is more effective and interpretable for cross-modal retrieval. The proposed method employs a two-stage strategy to fulfill the task. In the first stage, deep mutual information estimation is incorporated into the objective to maximize the mutual information between the input data and its embedding. In the second stage, an expelling branch is added to the network to disentangle the modality-exclusive information from the learned representations. This helps to reduce the impact of modality-exclusive information to the common subspace representation as well as improve the interpretability of the learned feature. Extensive experiments on two large-scale benchmark datasets demonstrate that our method can learn better visual-semantic embedding and achieve state-of-the-art cross-modal retrieval results.