Arrhythmias are prevalent cardiac disorders affecting millions worldwide. By analyzing cardiac motion modulated in mmWave reflections, mmWave sensing is emerging as a promising contactless revolution in arrhythmia detection compared to conventional contact-based methods. However, the fundamental bottleneck of existing mmWave sensing methods is their restriction to controlled laboratory settings with small-scale cohorts, limiting generalization to real-world populations. This limitation arises because mmWave signals undergo complex signal transformations during propagation, resulting in an explosion of signal diversity across large populations in real-world scenarios. Such diversity significantly complicates the direct recognition of arrhythmia. In this paper, we theoretically analyze the mechanism and impact of mmWave cardiac signal diversity. Leveraging the inherent transformation properties of mmWave signals, we propose a Diversity-Invariant Contrastive mmWave Sensing framework, which learns invariant features robust to complex signal transformations encountered in real-world scenarios. We evaluate our method in a practical, clinically-oriented scenario involving a large-scale population of 7,338 subjects, achieving an average F1-score of 0.8241 across four common arrhythmias. These results demonstrate that our method effectively bridges the diversity gap, representing a significant step toward practical clinical deployment of contactless arrhythmia detection via mmWave sensing.