Unsupervised cross-domain image retrieval aims to retrieve semantically consistent images across domains with significant domain gaps, which poses substantial challenges under the absence of annotations in both domains. Existing approaches primarily rely on internally derived supervision signals for representation learning and cross-domain alignment. However, such internally induced supervision tends to impose an upper bound on achievable retrieval performance, as it lacks stable semantic references to support reliable category-level correspondence across domains. To address these limitations, we propose Selective Constraint Learning (SCL), a framework that introduces external semantic guidance as a stable prior for unsupervised cross-domain image retrieval. Leveraging a pre-trained foundation model, SCL constructs a dual-scope constraint bank to capture high-confidence positive and negative semantic relations within and across domains. Based on this, we design a generic constraint loss to jointly facilitate intra-domain compactness and inter-domain alignment. In addition, prototypical geometry regularization is designed to enhance in-domain structural stability through prototype-centered pull-and-push forces. Extensive experiments on multiple benchmarks demonstrate that SCL consistently outperforms state-of-the-art methods.