Automatic modulation recognition is essential for large-scale wireless communications, but traditional methods often ignore unknown signals in open-set conditions, leading to their incorrect classification as known types and thereby compromising system reliability and communication security. To handle this challenge, a novel automatic modulation open set recognition (AMOSR) model based on deep metric learning with embedding enhancement is proposed in this article. First, for each known example, deep neural model is employed to separately extract the original in-phase and quadrature (IQ) signal and its instantaneous features, which are then fused to obtain embedding. Second, random erasing is employed to the original IQ signal and instantaneous features separately to obtain an augmented example, which has similar structure but different semantics with known example, and the embedding of this example is obtained by using first step. Then, the embedding space, in which tuplet loss and margin loss are combined with the embeddings of known and augmented examples, is trained to improve the overall performance of the model. Finally, after training, AMOSR is implemented using the class centers of known classes. Experiments on three automatic modulation datasets show that our model has better average performance than several mainstream methods in the field of computer vision.

Automatic Modulation Open Set Recognition (AMOSR) has practical significance in detecting unknown classes. However, a challenge arises when unknown samples closely resemble known samples, posing a formidable task for accurate detection. A novel AMOSR framework based on multi-view discriminators’ joint judgment is proposed to handle this challenge. Firstly, utilizing signal domain knowledge, multi-dimensional features are extracted through varied signal time-frequency transforms and encoders, baesd on which multiple discriminators are created. Secondly, Constrained Clustering Prototype Loss and Geodesic Contrastive Loss are introduced to pretrain these discriminators, providing more space for unknown signals. Then, collaborative learning is employed to further fine-tune the aforementioned discriminators, enhancing information sharing between modalities. Furthermore, a set of indicators is constructed, and multi-criteria fusion is performed using the TOPSIS algorithm to evaluate the discrimination capabilities of different classifiers in both closed-set and open-set scenarios. Furthermore, a decision tree is constructed to segregate test signals into known and unknown classes, in which discriminators with higher confidence levels are given precedence. Finally, TOPSIS hierarchical ensemble pruning algorithm that considers diversity and open-set recognition capabilities is adopted to reduce model complexity while maintaining original performance. Extensive experiments conducted on modulation datasets demonstrate the superiority of this framework over state-of-the-art AMOSR results.