In this paper, we present a novel task of source-free cross-modal adversarial example generation, which generates adversarial examples based on textual descriptions of attackers. This task has two challenges as follows. First, how to generate adversarial examples when the clean examples are missing or inaccessible. Second, how to achieve fine-grained custom adversarial example generation according to the semantic descriptions of the attackers. Existing adversarial example generation methods can not effectively deal with these two challenges. To address these challenges, we propose a Source-Free Cross-Modal Adversarial Example Generation framework, abbreviated as SFCM-AEG. Within the SFCM-AEG model, we firstly leverage a pre-trained GPT as a simulator to construct textual descriptions of attackers by labels. Following this, we employ a diffusion model to synthesize an image that aligns with the generated textual description. Finally, the generated images are converted into adversarial examples using an adversarial example generation method. Experimental results demonstrate that our proposed SFCM-AEG method can generate adversarial examples with customized semantic descriptions, without relying on clean examples, while achieving strong attack performance in a white-box setting.