Voice synthesis technologies have advanced rapidly, raising serious concerns about content security and trust. While many fake voice detectors achieve strong performance in controlled settings, they often overfit to speaker-irrelative features (SiFs), exhibit poor robustness, and fail in multi-speaker scenarios. To address these limitations, we propose MultiSiFer, a novel fake voice detector grounded in a new design philosophy: rather than merely distinguishing synthetic from human voices, it explicitly prioritizes learning essential human voice characteristics. MultiSiFer leverages a pre-trained speech representation model to enhance this learning and is the first detector trained on a newly curated multi-speaker fake voice dataset, enabling effective generalization across speakers. Experiments show that MultiSiFer outperforms existing methods in both standard and multi-speaker settings, achieving 10.84% average equal error rate (EER).

The application of deep learning in voice cloning has significantly enhanced the quality of cloned voices. While advanced voice cloning technologies are widely applied across various domains, they also pose serious security challenges such as producing natural Deepfakes. In response, numerous studies have focused on detecting fake voices, with many reporting outstanding performance. However, is the issue truly resolved? This paper introduces Adversarial Neural Mimicry Attack (ANMA) which leverages a specialized model to predict the behavior of other similar models, transforming black-box attacks into white-box scenarios indirectly. Based on ANMA and Speaker-irrelative Features (SiFs), we propose a novel black-box attack framework called SiFMimicEvader, designed to evade fake voice detectors with high success rates and minimal query requirements. The framework utilizes speech representation models as the breakthrough to predict the behaviors of fake voice detectors and employs a series of SiFs editing operations as perturbations to deceive these detectors. Experimental results demonstrate the effectiveness of SiFMimicEvader, achieving an average attack success rate exceeding 50% across various detectors, significantly outperforming other attack methods, while also showing great performance in audio quality and query scale, indicating its high availability in real-world scenarios.