The advent of voice cloning has fundamentally threatened the role of voice as a unique biometric. Many criminal incidents have already been reported to demonstrate its significant risks of identity forgery. Previous works explored the risks of voice cloning in constrained settings, which require victim speakers to either be already seen in the training data of voice cloning models, or leak dozens of minutes of their speech samples to adversaries. However, with the rapid progress of voice cloning in AIGC (Artificial Intelligence Generated Content) era, these requirements have largely been released, leaving the exact risks of state-of-the-art (SOTA) voice cloning techniques shrouded in a dense fog. To uncover it, this paper conducts a large-scale study in real-world scenarios to assess the risks of advanced voice cloning techniques. This study involves 5 SOTA voice cloning techniques (open-source and commercial), across 8 SOTA voice authentication systems (open-source and real-world) and 30 human listeners, using voice data of over 7,000 speakers (public and custom). By experimental and theoretical analysis, this study reveals that 1) state-of-the-art voice cloning techniques pose severe threats in spoofing voice authentication systems and human listeners; 2) demographic factors such as age and gender of victim speakers have a subtle impact on voice cloning attacks; 3) human listeners' subjective opinions and background about voice cloning play an important role in their susceptibility to attacks; 4) advanced detection methods still fail to identify voice cloning samples as expected.

As one of the most representative applications built on deep learning, audio systems, including keyword spotting, automatic speech recognition, and speaker identification, have recently been demonstrated to be vulnerable to adversarial examples, which have already raised general concerns in both academia and industry. Existing attacks follow the same adversarial example generation paradigm from computer vision, i.e., overlaying the optimized additive perturbations on original voices. However, due to the additive perturbations’ nature on human audibility, balancing the stealthiness and attack capability remains a challenging problem. In this paper, we rethink the stealthiness of audio adversarial examples and turn to introduce another kind of audio distortion, i.e., reverberation, as a new perturbation format for stealthy adversarial example generation. Such convolutional adversarial perturbations are crafted as real-world impulse responses and behave as a natural reverberation for deceiving humans. Based on this idea, we propose AdvReverb to construct, optimize, and deliver phoneme-level convolutional adversarial perturbations on both speech and music carriers with a well-designed objective. Experimental results demonstrate that AdvReverb could realize high attack success rates over 95% on three audio-domain tasks while achieving superior perceptual quality and keeping stealthy from human perception in over-the-air and over-the-line delivery scenarios.

Recent years have witnessed enormous research efforts on WiFi sensing to enable intelligent services of Internet of Things. However, due to the omni-directional broadcasting manner of WiFi signals, the activity semantic underlying the signals can be leaked to adversaries for surveillance, as demonstrated by our previous work. In this paper, we further extend the attack capability ofActListenerto impersonation attack, which could eavesdrop on users’ behavioral uniqueness imperceptibly using a WiFi infrastructure in any location of user sensing area. In particular,ActListener detects each human activityand converts the eavesdropped signals to that by legitimate devices based on our proposed signal propagation models. To extract noise-resilient individual behavioral uniqueness from converted CSI of WiFi signals, we further add user identification models into the substitute model set for training the signal pattern calibration generative model. Experimental results demonstrate thatActListenercould achieve over 80% accuracy in activity semantics retrieval and impersonation by using the converted signals.

With the broad integration of deep learning in Speaker Recognition (SR) systems, adversarial example attacks have been a significant threat raising user security concerns. Nevertheless, recent studies demonstrate that using input transformations (e.g., re-quantization, resampling, bandpass filtering) as a low-cost prefilter can efficiently mitigate such adversarial example attacks. These prefilters constrain the injection space of adversarial perturbations in both time and frequency domains, leading to either degraded attack performance or amplified perturbation noise. This paper proposes a new adversarial example attack, BypTalker, which could bypass these prefilter-enabled SR systems while remaining imperceptible to human listeners. BypTalker employs ensemble learning with diverse substitute pre-filters in the training phase to enhance the adversarial example’s adaptiveness to different prefilters. Furthermore, it incorporates an Acoustic Masker to cloak adversarial perturbations based on psychoacoustics effectively. This masker is well selected from a proposed metric M-Sup for minimizing the perturbation’s auditory to human perception. Experimental results show that BypTalker can achieve an Attack Success Rate of 99.1% and a Perceptual Evaluation of Speech Quality of 4.32, respectively.

Voice has become a fundamental method for human-computer interactions and person identification these days. Benefit from the rapid development of deep learning, speaker recognition exploiting voice biometrics has achieved great success in various applications. However, the shadow of adversarial example attacks on deep neural network-based speaker recognition recently raised extensive public concerns and enormous research interests. Although existing studies propose to generate adversarial examples by iterative optimization to deceive speaker recognition, these methods require multiple iterations to construct specific perturbations for a single voice, which is input-specific, time-consuming, and non-transferable, hindering the deployment and application for non-professional adversaries. In this paper, we propose PhoneyTalker, an out-of-the-box toolkit for any adversary to generate universal and transferable adversarial examples with low complexity, releasing the requirement for professional background and specialized equipment. PhoneyTalker decomposes an arbitrary voice into phone combinations and generates phone-level perturbations using a generative model, which are reusable for voices from different persons with various texts. Experiments on mainstream speaker recognition systems with large-scale corpus show that PhoneyTalker outperforms state-of-the-art methods with overall attack success rates of 99.9% and 84.0% under white-box and black-box settings respectively.

Faced with the threat of identity leakage during voice data publishing, users are engaged in a privacy-utility dilemma while enjoying convenient voice services. Existing studies employ direct modification or text-based re-synthesis to de-identify users' voices, but resulting in inconsistent audibility for human participants and not adaptive to informed attacks. In this poster, we propose a non-intrusive and adaptive speaker de-identification scheme to balance the privacy and utility of voice services. We generate adversarial examples to conceal user identity from exposure by Automatic Speaker Identification (ASI). By learning a compact distribution with a conditional variational auto-encoder, our system enables on-demand target sampling and diverse identity transformation. We also introduce the acoustic masking effect to construct inaudible perturbations, thus preserving the speech content and perceptual quality. Experiments on 50 speakers show our system could achieve 98.2% successful de-identification on 4 mainstream ASIs with an objective perceptual quality of 4.38 and a subjective mean opinion score of 4.56.

The integration of deep learning on Speaker Recognition (SR) advances its development and wide deployment, but also introduces the emerging threat of adversarial examples. However, only a few existing studies investigate its practical threat in physical domain, which either evaluate its feasibility only by directly replaying generated adversarial examples, or explore the partial channel interference for robustness improvement. In this paper, we propose a physical adversarial example attack, PhyTalker, which could generate and inject perturbations on voices in a live-streaming manner on attacking various SR models in different physical channels. Compared with the typical adversarial example for digital attacks, PhyTalker generates a subphoneme-level perturbation dictionary to decouple the perturbation optimization and injection. Moreover, we introduce the channel augmentation to compensate both device and environmental distortions, as well as model ensemble to improve the perturbation transferability. Finally, PhyTalker recognizes and localizes the latest recorded phoneme to determine the corresponding perturbations for real-time broadcasting. Extensive experiments are conducted with a large-scale corpus in real physical scenarios, and results show that PhyTalker achieves an overall Attack Success Rate (ASR) of 85.5% in attacking mainstream SR systems and Mel Cepstral Distortion (MCD) of 2.45dB in human audibility.