Recently, deep neural networks (DNNs) based emitter recognition or identification has received increasing interest. However, most of them are purely data-driven and require a large number of labeled instances. In this paper, a new Knowledge Driven Signal Transformer (KDSiT) is proposed, which introduces the knowledge graph (KG) into a signal Transformer (ST) model for accurate emitter recognition in real-world scenarios. On the one hand, KDSiT use a unified multimodal Transformer structure to explore the latent long-range dependencies in signals, and capture the subtle differences of emitters. On the other hand, KDSiT introduces domain knowledge, such as relationships and attributes between emitters, by constructing an emitter knowledge graph. By combining the powerful feature learning capability of DNNs with the rich semantic information in KG, KDSiT can extract more discriminative features of emitters from multimodal learning, to improve the identification accuracy in degraded environments. Extensive experiments are conducted, and the results prove the superiority of KDSiT over its counterparts, especially in the case of low signal-to-noise ratio (SNR), incomplete signals, and a limited number of labeled instances.