The adoption of augmented reality/virtual reality (AR/VR) has dramatically risen over the past few years across various application sectors, including immersive gaming, social communication, education, and tourism. The emerging use of AR/VR headsets has also created an excellent opportunity to promote pervasive health monitoring service as most AR/VR devices are already equipped with enriched sensing paradigm and will interact with users for a long time. In this talk, we aim to explore innovative technologies that enable fine-grained and personalized health status monitoring (e.g. vital signs and user identities) leveraging facial vibrations captured by the in-built motion sensor testbed on commodity AR/VR headsets. On one hand, it provides real-time health information required in virtual healthcare applications. For instance, a doctor can continuously monitor a patient's vital signs during the tele-medicine session at home, which helps the doctor to realize timely and precise diagnoses [2]. On the other hand, as people are spending increasing time in cyberspace (e.g., Metaverse), exposure to virtual and immersive contents requires high concentration on users' mind. Such usage cases may significantly increase the visual and psychological burden and induce potential health issues (e.g., anxiety, hypertension, sleep disorders) [1, 3, 5].

Despite the increasing popularity of voice assistants such as Amazon Alexa, the security implications of dynamic skill content (content modifiable without resubmission) in voice assistant skills (voice-activated applications) remain largely unexplored. This paper presents the first large-scale analysis of Alexa's dynamic content ecosystem using D-Explorer, a ChatGPT powered chatbot. From a dataset of 10,407 skill interactions, we investigate: 1) the mechanisms of Alexa dynamic content, 2) the associated security risks, and 3) the prevalence of these risks in published skills. Our analysis reveals that 34% of skills contain dynamic content in interactions, 95% access external resources (increasing attack vectors), 7% of skill conversations exhibit problematic (potentially harmful or privacy-infringing) interactions related to dynamic content, and 90% of skills connect to a potentially vulnerable dynamic resource during interaction. These findings expose significant vulnerabilities, highlighting the critical need for stricter developer rules and security measures to prevent unpredictable, harmful, and privacy compromising interactions within the Alexa skill ecosystem.

Vital signs (e.g., breathing and heart rates) and personal identities are essential information for personalized medicine and healthcare. The popularity of augmented reality/virtual reality (AR/VR) provides an excellent opportunity for enabling long-term health monitoring in a broad range of scenarios, including virtual entertainment, education, and telemedicine. However, commercial-off-the-shelf AR/VR devices do not have dedicated biosensors for providing vital signs and personal identities. In this work, we propose a novel framework that can generate fine-grained vital sign signals and other personalized health information of an AR/VR user through passive sensing on AR/VR devices. In particular, we find that the user's minute facial vibrations induced by breathing and heart beating can impact the readily available motion sensors on AR/VR headsets, which encode rich vital sign patterns and unique biometrics. The proposed framework further estimates the breathing and heartbeat rates, detects the gender and identity, and derives the body fat percentage of the user. To mitigate the impacts of body movement, we design an adaptive filtering scheme to cancel the spontaneous and non-spontaneous motion artifacts. We also develop unique facial vibration features and deep learning techniques to facilitate vital sign signal reconstruction and user identification. Extensive experiments demonstrate that our framework can achieve a low error of vital sign signal reconstruction and rate measurement, along with 95.51% and 93.33% accuracy on identity and gender recognition.

Virtual Reality (VR) has gained popularity in numerous fields, including gaming, social interactions, shopping, and education. In this paper, we conduct a comprehensive study to assess the trustworthiness of the embedded sensors on VR, which embed various forms of sensitive data that may put users’ privacy at risk. We find that accessing most on-board sensors (e.g., motion, position, and button sensors) on VR SDKs/APIs, such as OpenVR, Oculus Platform, and WebXR, requires no security permission, exposing a huge attack surface for an adversary to steal the user’s privacy. We validate this vulnerability through developing malware programs and malicious websites and specifically explore to what extent it exposes the user’s information in the context of keystroke snooping. To examine its actual threat in practice, the adversary in the considered attack model doesn’t possess any labeled data from the user nor knowledge about the user’s VR settings. Extensive experiments, involving two mainstream VR systems and four keyboards with different typing mechanisms, demonstrate that our proof-of-concept attack can recognize the user’s virtual typing with over 89.7% accuracy. The attack can recover the user’s passwords with up to 84.9% recognition accuracy if three attempts are allowed and achieve an average of 87.1% word recognition rate for paragraph inference. We hope this study will help the community gain awareness of the vulnerability in the sensor management of current VR systems and provide insights to facilitate the future design of more comprehensive and restricted sensor access control mechanisms.

The growing adoption of voice-enabled devices (e.g., smart speakers), particularly in smart home environments, has introduced many security vulnerabilities that pose significant threats to users' privacy and safety. When multiple devices are connected to a voice assistant, an attacker can cause serious damage if they can gain control of these devices. We ask where and how can an attacker issue clean voice commands stealthily across a physical barrier, and perform the first academic measurement study of this nature on the command injection attack. We present the BarrierBypass attack that can be launched against three different barrier-based scenarios termed across-door, across-window, and across-wall. We conduct a broad set of experiments to observe the command injection attack success rates for multiple speaker samples (TTS and live human recorded) at different command audio volumes (65, 75, 85 dB), and smart speaker locations (0.1-4.0m from barrier). Against Amazon Echo Dot 2, BarrierBypass is able to achieve 100% wake word and command injection success for the across-wall and across-window attacks, and for the across-door attack (up to 2 meters). At 4 meters for the across-door attack, BarrierBypass can achieve 90% and 80% injection accuracy for the wake word and command, respectively. Against Google Home mini BarrierBypass is able to achieve 100% wake word injection accuracy for all attack scenarios. For command injection BarrierBypass can achieve 100% accuracy for all the three barrier settings (up to 2 meters). For the across-door attack at 4 meters, BarrierBypass can achieve 80% command injection accuracy. Further, our demonstration using drones yielded high command injection success, up to 100%. Overall, our results demonstrate the potentially devastating nature of this vulnerability to control a user's device from outside of the device's physical space, and its limitations, without the need for complex and error-prone command injection.

Sound waves from speech can potentially induce vibrations, proportional to the speech signal, on nearby objects. Each of these objects introduces the risk for a malicious attacker to exploit the induced vibrations to eavesdrop on the speech. Such an eavesdropping attack is critical when we consider the potential for induced vibrations in standard magnetic hard disk drives (HDDs). As an instance of this threat, prior research has demonstrated that speech in certain scenarios can induce vibrations on the read/write head of an HDD in order to eavesdrop on the speech (Kwong et al.; Oakland'19). In this paper, we revisit this line of research and aim to provide a closer investigation into whether HDDs can in fact be used as a source for eavesdropping on speech vibrations. As a foundation for our study, we utilize an effective, and robust methodology using laser vibrometry to measure the subtle speech vibrations induced on the read/write head. The prior study tested only a single HDD and only machine-rendered speech in a single setting with very loud speech. Our work broadens the scope of this research in many significant ways. First, we test multiple popular HDDs of different models and sizes to evaluate the generalizability of the overall threat. Second, we evaluate the threat from live human speech spoken near an HDD, expanding the scope of the attack to include most real-world speech settings involving normal human conversations. Third, we define machine-rendered speech scenarios to explore different propagation media and degrees of speech loudness. Our findings are two-fold. First, we observed that live human speech traveling through the air is not generally strong enough to impact HDDs such that intelligible speech information is leaked. Second, most tested HDDs did not seem capable of eavesdropping on machine-rendered speech unless the speech is loud enough, or the HDD shares a surface or is in direct contact with the speaker device. This implies HDDs cannot eavesdrop live human speech.

Human speech signals produce sound waves that induce vibrations on objects that they encounter. Such vibrations can be measured via laser vibrometers and possibly used in speech eavesdropping attacks. However, there is still much to learn about when this attack is feasible. In this paper, we aim to broaden our understanding of the viability of laser eavesdropping attacks to compromise speech in the commercial user space. In our study, we design experiments to measure the subtle vibrations induced on commonly-available objects by nearby speech, using commercially sold, high-precision laser vibrometers. To observe idealized success rates of the attack, we maintain certain physical parameters in favorable conditions that represent best case scenarios for an attacker. We test three primary attack scenarios considering different relative positions to the target object. Additionally, we consider many important experimental parameters to understand the generalizability of the attack, including: speech sources, loudness levels, vibration propagation media, and object materials. Our vibrometer recorded signals were analyzed via a two-pronged methodology including, (1) time domain, frequency spectrum, cross correlation, and speech intelligibility metric analyses and (2) an information extraction analysis using both human listeners and automated recognition tools. Our results suggest that eavesdropping attacks using a laser vibrometer may be practical in some situations and parameter settings (i.e., intelligence missions). However, we find that live aerial human speech and machine-rendered speech at a normal conversational loudness level does not show signs of significant leakage in our analysis.

Augmented reality/virtual reality (AR/VR) headsets have attracted millions of users and gained predictable popularity. However, long-period usage of immersive technology may lead to health issues (e.g., cybersickness, anxiety). In this poster, we design a low-cost and personalized healthcare monitoring system grounded on vital sign tracking (i.e., breathing and heartbeat rate tracking), by exploiting built-in AR/VR motion sensors. The key insight is that the conductive vibrations induced by chest and heart movements can propagate through the user's cranial bones, thereby vibrating the AR/VR headset mounted on the user's head. To realize this system, we design signal processing techniques to cancel the human motions and derive the periods of breathing and heartbeat through frequency-domain analyses. We further design a user identification scheme based on respiratory and cardiac biometrics, which works with vital sign monitoring to provide personalized healthcare recommendations. Our experiment shows that the proposed scheme can achieve less than 5.7% error rate on breathing/heartbeat rate estimation and 95% accuracy on user identification.

Modern speech recognition systems that are widely deployed today still suffer from known gender and racial biases. In this work, we demonstrate the potential to exploit the existing biases in these systems to achieve a new attack goal. We consider the potential for command disruption by an attacker that can be conducted in a manner that allows for access and control of a victim's voice assistant device. We present a novel attack, BiasHacker, which crafts specialized chatter noise to exploit racial and gender biases in speech recognition systems for the purposes of command disruption. Our experimental results confirm both racial and gender bias that is still present in the speech recognition systems of two modern smart speaker devices. We also evaluated the effectiveness of three types of chatter noise (American English (AE)-Male, Nigerian-Female, Korean-Female) for disruption and demonstrate that the AE-Male chatter is consistently more successful. Comparing the average success rate of each chatter type, in scenarios where disruption was achieved, we find that when targeting the Google Home mini smart speaker, the AE-Male chatter noise increases average disruption success compared to the Nigerian-Female and Korean-Female chatter noises by 112% and 121%, respectively. Also, when targeting the Amazon Echo Dot 2 the AE-Male chatter noise increases average disruption success compared to the Nigerian-Female and Korean-Female chatter noises by 42% and 69%, respectively.

Protection Jamming Devices (PJD) are specialized tools designed to sit on top of virtual assistant (VA) smart speakers and hinder them from “hearing” nearby user speech. PJDs aim to protect you from eavesdropping attacks by injecting a jamming signal directly into the microphones of the smart speaker. However, current signal processing routines can be used to reduce noise and enhance speech contained in noisy audio samples. Therefore, we identify a potential vulnerability for speech eavesdropping via smart speaker recordings, even when a PJD is being used. If an attacker can gain access to or facilitate smart speaker recordings they may be able to compromise a user’s speech with successful noise cancellation. Specifically, we are interested in the potential for Gaussian white noise (GWN) to be an effective jamming signal for a PJD. To our knowledge, the effectiveness of white noise and PJDs to protect against eavesdropping attacks has yet to receive a systematic evaluation that includes physical experiments with an actual PJD implementation.

Recent years have witnessed the rapid development of automatic speech recognition (ASR) systems, providing a practical voice-user interface for widely deployed smart devices. With the ever-growing deployment of such an interface, several voice-based attack schemes have been proposed towards current ASR systems to exploit certain vulnerabilities. Posing one of the more serious threats,hidden voice attack uses the human-machine perception gap to generate obfuscated/hidden voice commands that are unintelligible to human listeners but can be interpreted as commands by machines. However, due to the nature of hidden voice commands (i.e., normal and obfuscated samples exhibit a significant difference in their acoustic features), recent studies show that they can be easily detected and defended by a pre-trained classifier, thereby making it less threatening. In this paper, we validate that such a defense strategy can be circumvented with a more advanced type of hidden voice attack calledHVAC. Our proposed HVAC attack can easily bypass the existing learning-based defense classifiers while preserving all the essential characteristics of hidden voice attacks (i.e., unintelligible to humans and recognizable to machines). Specifically, we find that all classifier-based defenses build on top of classification models that are trained with acoustic features extracted from the entire audio of normal and obfuscated samples. However, only speech parts (i.e., human voice parts) of these samples contain the useful linguistic information needed for machine transcription. We thus propose a fusion-based method to combine the normal sample and corresponding obfuscated sample as a hybrid HVAC command, which can effectively cheat the defense classifiers. Moreover, to make the command more unintelligible to humans, we tune the speed and pitch of the sample and make it even more distorted in the time domain while ensuring it can still be recognized by machines. Extensive physical over-the-air experiments demonstrate the robustness and generalizability of our HVAC attack under different realistic attack scenarios. Results show that our HVAC commands can achieve an average 94.1% success rate of bypassing machine-learning-based defense approaches under various realistic settings.

Augmented reality/virtual reality (AR/VR) has extended beyond 3D immersive gaming to a broader array of applications, such as shopping, tourism, education. And recently there has been a large shift from handheld-controller dominated interactions to headset-dominated interactions via voice interfaces. In this work, we show a serious privacy risk of using voice interfaces while the user is wearing the face-mounted AR/VR devices. Specifically, we design an eavesdropping attack, Face-Mic, which leverages speech-associated subtle facial dynamics captured by zero-permission motion sensors in AR/VR headsets to infer highly sensitive information from live human speech, including speaker gender, identity, and speech content. Face-Mic is grounded on a key insight that AR/VR headsets are closely mounted on the user's face, allowing a potentially malicious app on the headset to capture underlying facial dynamics as the wearer speaks, including movements of facial muscles and bone-borne vibrations, which encode private biometrics and speech characteristics. To mitigate the impacts of body movements, we develop a signal source separation technique to identify and separate the speech-associated facial dynamics from other types of body movements. We further extract representative features with respect to the two types of facial dynamics. We successfully demonstrate the privacy leakage through AR/VR headsets by deriving the user's gender/identity and extracting speech information via the development of a deep learning-based framework. Extensive experiments using four mainstream VR headsets validate the generalizability, effectiveness, and high accuracy of Face-Mic.

Existing academic research on vibration-based speech attacks has introduced interesting and intellectually appealing threat vectors with proof-of-concept demonstrations in controlled environments. The attacks presented in these studies exploit different types of sensors such as MEMS motion sensors, laser-based sensors, and some other sensors (camera, position error signal, piezo-disc) to measure the vibrations induced on an object by nearby sensitive speech. Such sensors are commonly found on mobile devices like smartphones and tablets that can be exposed to sensitive speech, revealing the significance of this potential threat. These studies have amassed significant attention in news and media and introduced concern to people about the safety of their day-to-day speech and around their personal, wireless and IoT devices. However, we hypothesize that the controlled experiments in the prior research maintain critical parameter values that are favorable to attack success (deviating from the limiting settings in a real-world scenario) and produce results that suggest a greater real-life threat level than actually exists.

Voice access technologies are widely adopted in mobile devices and voice assistant systems as a convenient way of user interaction. Recent studies have demonstrated a potentially serious vulnerability of the existing voice interfaces on these systems to "hidden voice commands". This attack uses synthetically rendered adversarial sounds embedded within a voice command to trick the speech recognition process into executing malicious commands, without being noticed by legitimate users.

This paper explores the effectiveness of common sound masking solutions deployed for preserving speech privacy in workplace environment such as hospitals, financial institutions, lawyers offices, nursing homes and government buildings. With the increased awareness about personal privacy among the general population, we set out to examine the effectiveness of current speech privacy preserving tools. We seek to determine if the general approach used by the current masking mechanisms is adequate to provide the level of privacy desired from these solutions. In addition, we also seek to investigate preservation of speech privacy in the face of ubiquitous and less conspicuous devices like smartphones that possess the capability of sound recording with inbuilt noise cancellation technology. Our approach in this paper is to expose the vulnerability in sound masking technology in scenarios that require preserving privacy in personal spaces. We use human listeners to attack speech privacy under sound masking where we aim to identify spoken words eavesdropped under different scenarios. We also test currently available speech recognition tools to assess their performance at decoding speech in noisy environment. Our results indicate that pink noise, the commonly used technology to provide speech privacy for use in personal space, is ineffective against a dedicated eavesdropping adversary that uses commonplace devices such as smartphones to record the speech and noise reduction tools to counteract sound masking.