Cross-modal retrieval techniques, with their efficient search capabilities, have garnered considerable attention in both industry and academia. Among them, image-text cross-modal retrieval, a classic task in the cross-modal domain, has been widely applied in internet applications. However, image data in cross-modal retrieval applications often contains rich personal sensitive information, which is typically stored in plaintext format, posing a risk of privacy leakage. Due to the unique semantic and heterogeneous gap in the cross-modal space, existing single-modality image privacy protection techniques cannot be directly applied to cross-modal scenarios. To address this issue, this paper proposes a privacy-preserving cross-modal retrieval method based on disentangled soft-label alignment(PPDSA). Firstly, to protect the privacy of image data, thumbnail preserving encryption is used in this paper. This method aims to encrypt the image while ensuring the feasibility of graph-text cross-modal feature representation learning, so as to realize cross-modal encrypted retrieval. Second, to further improve the retrieval effect of encrypted images and normal text, this paper introduces a disentangled soft label alignment technique. Specifically, a teacher model is used to obtain the soft label matrix, and cross-modal (encrypted image-text pair) and single-modal (encrypted image pair) soft label alignment techniques are designed to capture more fine-grained semantic representation information and reduce the interference of false positives on similarity recognition. A series of experiments on Flickr30k and MSCOCO datasets show that the proposed method not only effectively protects image privacy in cross-modal retrieval, but also improves the retrieval performance by 6.7% and 3.7% respectively compared with the CLIP baseline.