In the context of zero-shot learning, retrieving natural images using sketch queries is referred to as Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR). The core problems of ZS-SBIR arise from the significant domain gap between sketches and photographs. The knowledge gap between seen and unseen categories is a crucial problems as well. To address these two issues, we propose a modality-aware bidirectional mining strategy aimed at enhancing the model’s understanding of multimodal data and mitigating the modality gap between sketches and photographs. Additionally, we introduce an improved Mixed Sample Data Augmentation (MSDA) technique to generate mixed samples as supplementary training data. Based on these generated mixed samples, we propose a mixed embedding mining strategy. This strategy aims to optimize the embedding space and enhance the model’s generalization ability from seen to unseen categories. Extensive experimental results on the TU-Berlin Ext., Sketchy Ext., and QuickDraw Ext. datasets demonstrate that our proposed model outperforms existing methods.

Sketch-based image retrieval (SBIR) relies on free-hand sketches to retrieve natural photos within the same class. However, its practical application is limited by its inability to retrieve classes absent from the training set. To address this limitation, the task has evolved into Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR), where model performance is evaluated on unseen categories. Traditional SBIR primarily focuses on narrowing the domain gap between photo and sketch modalities. However, in the zero-shot setting, the model not only needs to address this cross-modal discrepancy but also requires a strong generalization capability to transfer knowledge to unseen categories. To this end, we propose a novel framework for ZS-SBIR that employs a pair-based relation-aware quadruplet loss to bridge feature gaps. By incorporating two negative samples from different modalities, the approach prevents positive features from becoming disproportionately distant from one modality while remaining close to another, thus enhancing inter-class separability. We also propose a Relation-Aware Meta-Learning Network (RAMLN) to obtain the margin, a hyper-parameter of cross-modal quadruplet loss, to improve the generalization ability of the model. RAMLN leverages external memory to store feature information, which it utilizes to assign optimal margin values. Experimental results obtained on the extended Sketchy and TU-Berlin datasets show a sharp improvement over existing state-of-the-art methods in ZS-SBIR.