As Augmented Reality (AR) devices become more prevalent and commercially viable, the need for quick, efficient, and secure schemes for pairing these devices has become more pressing. Current methods to securely exchange holograms require users to send this information through large data centers, creating security and privacy concerns. Existing techniques to pair these devices on a local network and share information fall short in terms of usability and scalability. These techniques either require hardware not available on AR devices, intricate physical gestures, removal of the device from the head, do not scale to multiple pairing partners, or rely on methods with low entropy to create encryption keys. To that end, we propose a novel pairing system, called GazePair, that improves on all existing local pairing techniques by creating an efficient, effective, and intuitive pairing protocol. GazePair uses eye gaze tracking and a spoken key sequence cue (KSC) to generate identical, independently generated symmetric encryption keys with 64 bits of entropy. GazePair also achieves improvements in pairing success rates and times over current methods. Additionally, we show that GazePair can extend to multiple users. Finally, we assert that GazePair can be used on any Mixed Reality (MR) device equipped with eye gaze tracking.

Augmented Reality (AR) devices are set apart from other mobile devices by the immersive experience they offer. While the powerful suite of sensors on modern AR devices is necessary for enabling such an immersive experience, they can create unease in bystanders (i.e., those surrounding the device during its use) due to potential bystander data leaks, which is called the bystander privacy problem. In this paper, we propose BystandAR, the first practical system that can effectively protect bystander visual (camera and depth) data in real-time with only on-device processing. BystandAR builds on a key insight that the device user's eye gaze and voice are highly effective indicators for subject/bystander detection in interpersonal interaction, and leverages novel AR capabilities such as eye gaze tracking, wearer-focused microphone, and spatial awareness to achieve a usable frame rate without offloading sensitive information. Through a 16-participant user study,we show that BystandAR correctly identifies and protects 98.14% of bystanders while allowing access to 96.27% of subjects. We accomplish this with average frame rates of 52.6 frames per second without the need to offload unprotected bystander data to another device.

Augmented Reality (AR) devices are set apart from other mobile devices by the immersive experience they offer. While the powerful suite of sensors on modern AR devices is necessary for enabling such an immersive experience, they can create unease in bystanders (i.e., those surrounding the device during its use) due to potential bystander data leaks, which is called the bystander privacy problem. In this poster, we propose BystandAR, the first practical system that can effectively protect bystander visual (camera and depth) data in real-time with only on-device processing. BystandAR builds on a key insight that the device user's eye gaze and voice are highly effective indicators for subject/bystander detection in interpersonal interaction, and leverages novel AR capabilities such as eye gaze tracking, wearer-focused microphone, and spatial awareness to achieve a usable frame rate without offloading sensitive information. Through a 16-participant user study, we show that BystandAR correctly identifies and protects 98.14% of bystanders while allowing access to 96.27% of subjects. We accomplish this with average frame rates of 52.6 frames per second without the need to offload unprotected bystander data to another device.

Augmented reality (AR) applications that overlay the perception of the real world with digitally generated information are on the cusp of commercial viability. AR has appeared in several commercial platforms like Microsoft HoloLens and smartphones. They extend the user experience beyond two dimensions and supplement the normal 3D world of a user. A typical location-based multi-player AR application works through a three-step process, wherein the system collects sensory data from the real world, identifies objects based on their context, and finally, renders information on top of senses of a user. However, because these AR applications frequently exchange data with users, they have exposed new individual and public safety issues. In this paper, we develop ARSpy, a user location tracking system solely based on network traffic information of the user, and we test it on location-based multi-player AR applications. We demonstrate the effectiveness and efficiency of the proposed scheme via real-world experiments on 12 volunteers and show that we could obtain the geolocation of any target with high accuracy. We also propose three mitigation methods to mitigate these side channel attacks. Our results reveal a potential security threat in current location-based multi-player AR applications and serve as a critical security reminder to a vast number of AR users.

Voice-based input is usually used as the primary input method for augmented reality (AR) headsets due to immersive AR experience and good recognition performance. However, recent researches show that attackers can inject inaudible voice commands to the devices that lack voice verification. Even if we secure voice input with voice verification techniques, attackers can record the victim’s voice and replay it. To defend against voice-spoofing attacks, AR headsets should be able to determine whether the voice is from the person who is using the AR headsets. Existing voice-spoofing defense systems are designed for smartphone platforms and usually fail to work due to the special locations of microphones and loudspeakers on AR headsets. To address this challenge, in this paper, we propose a voice-spoofing defense system for AR headsets by leveraging both the internal body propagation and the air propagation of human voices. Experimental results show that our system can successfully accept normal users with average accuracy of 97 percent and defend against two basic types of attacks with average accuracy of at least 98 percent. More importantly, even if the attackers can fool our line-fitting model by manipulating special voice signals, our MCD-SVDD model can still reject them with accuracy of 100 percent.

With the rapid popularity of cameras on various devices, video chat has become one of the major ways for communication, such as online meetings. However, the recent progress of face reenactment techniques enables attackers to generate fake facial videos and use others' identities. To protect video chats against fake facial videos, we propose a new defense system to significantly raise the bar for face reenactment-assisted attacks. Compared with existing works, our system has three major strengths. First, our system does not require extra hardware or intense computational resources. Second, it follows the normal video chat process and does not significantly degrade the user experience. Third, our system does not need to collect training data from attackers and new users, which means it can be quickly launched on new devices. We developed a prototype and conducted comprehensive evaluations. Experimental results show that our system can provide an average true acceptance rate of at least 92.5% for legitimate users and reject the attacker with mean accuracy of at least 94.4% for a single detection.

With the success of voice recognition techniques, users can easily control any device in smart home environments by simply saying a voice command. Based on this idea, a new group of smart devices are designed and released, which are called voice assistant. However, the voice itself is not secure and can be attacked in many ways. To defend against various types of voice replay attacks, we present a new voice liveness detection system. The basic insight of our system is that mouth opening movements will change the space size in the ear canal, which further changes the air pressure in ear canals. In this paper, we propose solutions to detect mouth opening movements using the noisy air pressure data and match them with the voices to validate the liveness of the voice source. To evaluate the effectiveness of our system, we develop a prototype on Raspberry Pi and conduct comprehensive evaluations. Experiments with ten volunteers show that our system can accurately accept voice commands from legitimate users with an accuracy of 91.72%. Moreover, our system can effectively defend current voice assistant devices from replay attacks with an accuracy of 97.2%.

Nowadays, personal identification number (PIN) is one of the most popular methods for identity verification. However, recent researches show that attackers can easily recover victims’ PINs in spite of the large number of combinations PIN provides. Existing protection approaches require alteration of the original interaction between the user and PIN-based authentication systems, or still fail if the attacker can observe and mimic the victim’s input behavior. Considering these limitations, we propose a defense system called LightDefender to protect current PIN-based systems from PIN replay attacks using a single ambient light sensor. Specifically, we protect the PIN input by leveraging the biometrics in the received light intensity that is influenced by input behaviors and biological features. To our best knowledge, our work is the first one to protect PIN input using the light intensity. Different from existing approaches, LightDefender does not change the original interaction methods between the user and PIN-based authentication systems, and the extra hardware cost is low. In addition, by leveraging biological differences (e.g. finger length) among different users, LightDefender still claims high-security protection against strong attackers who can mimic the victim’s input behaviors. Experiments with 10 volunteers show that LightDefender can achieve an average true acceptance rate of 95% for normal users. More importantly, LightDefender can correctly reject two types attackers with an average true rejection rate of at least 93.6% without data of new attackers.

Voice-based input is usually used as the primary input method for augmented reality (AR) headsets due to immersive AR experience and good recognition performance. However, recent researches have shown that an attacker can inject inaudible voice commands to the devices that lack voice verification. Even if we secure voice input with voice verification techniques, an attacker can easily steal the victim's voice using low-cast handy recorders and replay it to voice-based applications. To defend against voice-spoofing attacks, AR headsets should be able to determine whether the voice is from the person who is using the AR headsets. Existing voice-spoofing defense systems are designed for smartphone platforms. Due to the special locations of microphones and loudspeakers on AR headsets, existing solutions are hard to be implemented on AR headsets. To address this challenge, in this paper, we propose a voice-spoofing defense system for AR headsets by leveraging both the internal body propagation and the air propagation of human voices. Experimental results show that our system can successfully accept normal users with average accuracy of 97% and defend against two types of attacks with average accuracy of at least 98%.

While modern wireless devices are capable of using multiple WiFi interfaces, the Multipath TCP (MPTCP) protocol has been employed to make full use of the capacity of many radios by enabling multiple path communication simultaneously. To provide exceptional mobility support in wireless networks, a key question is to determine the best handoff strategy to switch between access points or among WiFi/3G interfaces during roaming. In this paper, we propose SERO, a novel model-driven SEamless ROaming framework to optimize layer-2 handoff and vertical handoff for multihomed devices using MPTCP. The proposed framework adopts a measurement-based method to derive the TCP throughput model for wireless communication during handoff. Based on the throughput model, we propose a hybrid handoff strategy that uses multiple WiFi interfaces for data transmission and employs 3G augmentation to bridge the network interruption caused by handoff and to guarantee the total throughput above a predefined threshold for roaming devices. We implement the SERO framework in a real-deployed wireless mesh network testbed, and evaluate its performance by extensive experiments, which shows that SERO achieves performance gain of 26%-180% compared with several existing handoff strategies.

Voiceprint-based authentication is fast becoming the everyday norm since it is much easier to use and provides better security. However, current voiceprint-based authentication systems are vulnerable to various replay attacks. To tackle the spoofing attacks, we propose a new system that leverages the structural differences between human vocal system and loudspeakers and use the unique vibration pattern of both human vocal cord and throat as a key differentiating factor for liveness detection. Specially, we model the relationship between voices collected by two microphones of a smartphone of each live speaker using sparse representation. Compared with existing systems, our solution does not assume any prior knowledge of the attack method and is easy to operate. Moreover, our solution leverages the audio signals within the vocal frequency range and is robust to jamming attacks using high-frequency audio. Experimental results show that our system can achieve accurate live ness detection for a 6-digit passphrase with a mean true acceptance rate of 99.04% and true rejection rate of 100%.

Mobile Augmented Reality (AR) applications allow the user to interact with virtual objects positioned within the real world via a smart phone, tablet or smart glasses. As the popularity of these applications grows, recent researchers have identified several security and privacy issues pertaining to the collection and storage of sensitive data from device sensors. Location-based AR applications typically not only collect user location data, but transmit it to a remote server in order to download nearby virtual content. In this paper we show that the pattern of network traffic generated by this process alone can be used to infer the user's location. We demonstrate a side-channel attack against a widely available Mobile AR application inspired by Website Fingerprinting methods. Through the strategic placement of virtual content and prerecording of the network traffic produced by interacting with this content, we are able to identify the location of a user within the target area with an accuracy of 94%. This finding reveals a previously unexplored vulnerability in the implementation of Mobile AR applications and we offer several recommendations to mitigate this threat.

The recent proliferation of smartphones has been the primary driving factor behind the booming of voice-based mobile applications. However, the human voice is often exposed to the public in many different scenarios, and an adversary can easily "steal" a person's voice and attack voice-based applications with the help of state-of-the-art voice synthesis/conversion softwares. In this paper, we propose a robust software-based voice liveness detection system for defending against voice spoofing attack. The proposed system is tailored for mobile platforms and can be easily integrated with existing mobile applications. We propose three approaches based on leveraging the vibration of human vocal cords, the motion of the human vocal system, and the functionality of vibration motor inside the smartphone. Experimental results show that our system can detect a live speaker with a mean accuracy of 94.38% and detect an attacker with a mean accuracy of 88.89% by combining three approaches we proposed.

Sign language is important since it permits insight into the deaf culture and allows more opportunities to communicate with those who are deaf or hard of hearing. In this paper, we show that Wi-Fi signals can be used to recognize sign language with sparsely labeled training dataset. The key intuition is that sign language introduces different multi-path distortions in Wi-Fi signals and generates different unique patterns in the time-series of Channel State Information (CSI) values. Based on these observations, we propose a sign language recognition system called WiSign. Different from existing Wi-Fi signal-based human activity recognition systems, WiSign only requires a sparsely labeled training dataset. Two solutions based on transfer learning and semi-supervised learning are proposed to reduce the number of required labeled instances. We implemented WiSign using a TP-Link TL-WR1043ND Wi-Fi router and a Lenovo X100e laptop. The evaluation results show that WiSign can achieve a mean prediction accuracy of 87.01% and 87.38% for the transfer learning-based approach and semi-supervised learning-based approach, respectively.

We have proposed a new form of action recognition using motion sensors built in mobile and wearable devices. Due to the miniaturization of hardware sensors, action classification through mobile sensors has become a much more attainable task. Using Android and Tizens integrated development environment, we have devised applications for each of these devices to document raw sensor data for analysis. Utilizing dynamic time warping, we attempt to recognize and classify actions based on differences in euclidean distances to build a strong database for further development.

Wireless mesh network (WMN) has been proposed to extend wireless coverage by using multi-hop communication to enable network service for mobile devices via densely deployed mesh APs. While modern wireless devices are capable of using multiple WiFi interfaces, the Multipath TCP (MPTCP) protocol has been employed to make full use of the capacity of multiple radios. A key question of MPTCP communication in WMN is to decide the best access point (AP) association for each wireless interface to achieve maximum network performance. In this paper, we propose an optimal AP association scheme based on goodput measurement. We setup a wireless mesh testbed with densely deployed APs to study the impact of signal strength and multi-hop communication. We fit the measured data by a three-dimensional surface, which shows that network goodput can be estimated by a function of Received Signal Strength Indicator (RSSI) and hops. With the proposed estimation function, we formulate the AP association problem as a mathematical optimization problem and solve it by Integer Programming.We implement the proposed strategy in the real deployed testbed and evaluate its performance using MPTCP, which shows that it improves MPTCP goodput significantly compared to several existing strategies.