Unlike static biometrics (e.g., faces and fingerprints), dynamic behavioral biometrics are believed to be more difficult to replicate. This paper investigates the security of behavioral biometrics considering the advancements in robotics and AI, particularly as humanoid robots, like Tesla Optimus, are expected to be mass-produced in the coming years. We find that general robotic arms have already gained the capability to reproduce human hand motion trajectories. However, using robots to replicate a user’s behavioral biometrics for attacks remains under-explored due to two long-standing challenges: 1) how to obtain the user’s complex behavioral biometrics through practical eavesdropping (not just trajectories); 2) how to replicate the user’s behavioral kinematics based on the eavesdropped data using a real robot. This work is the first to comprehensively address the two challenges. We develop the point-wise GAN-based Robot Replay Attack (GANRRA) to demonstrate a practical human behavioral replay attack using a hidden camera and a physical robot. GANRRA utilizes a hidden camera to eavesdrop on the user’s hand motions and employs a generative adversarial network to reconstruct the motion data, addressing the sensor discrepancies between the legitimate sensor and the hidden camera and maximizing the behavioral feature similarities. The reconstructed motion data is converted into velocity commands for a robot to execute point by point, replicating both hand movement trajectories and behavioral biometric features. For experiments, we implement an in-air signature system using two existing hand-tracking systems and fool them using a robotic arm attached with a fake hand. Results show that GANRRA reproduces in-air signatures with a 73.1% success rate. To address such robot-relay threats, a novel defense mechanism based on multi-joint behaviors is proposed.

Teleoperated robotics, which translates human behavior into robotic actions, remains a critical area of modern robotics. Although autonomous systems have advanced rapidly, they still struggle in complex and unstructured environments, making human-in-the-loop control indispensable for many real-world tasks. Teleoperation platforms commonly rely on motion-tracking technologies to capture detailed operator behavior, which is subsequently converted into robot control commands. However, these rich behavioral signals can also encode operator-specific biometrics, posing privacy risks such as user re-identification. While prior work shows that behavioral biometrics can be leveraged for reliable authentication, privacy leakage in teleoperation-centric motion streams has received comparatively less attention. To address this gap, we introduce a disentangled representation-learning framework based on a Variational Autoencoder (VAE) to suppress identity-revealing cues while retaining task-relevant motion patterns. We evaluate the proposed approach offline on reconstructed trajectories collected from a tele-robotic prototype, where multiple users perform a set of manipulation tasks. Our results demonstrate a substantial reduction in re-identification risk and a favorable privacy–utility trade-off in terms of task utility. More broadly, our findings highlight the need for robust privacy protections in future robotic teleoperation systems.

The increasing reliance on smartphones for secure digital interactions requires robust, user-friendly authentication methods. Current smartphone authentication depends on dedicated biometric sensors (e.g., fingerprint or facial recognition), which not only raise hardware costs but are also subject to privacy concerns. Meanwhile, most existing methods overlook the potential of passive, natural interactions like tapping the screen with vibration feedback as opportunities for seamless and secure authentication using only built-in smartphone sensors. This paper proposes a versatile behavioral biometric authentication method that leverages active motor vibration during screen taps to verify user identity. To our knowledge, this is the first system to demonstrate that standard touchscreens can capture fingerprint-like biometrics. Specifically, when a user interacts with the device, brief vibrations generated by the smartphone’s haptic motor elicit distinctive user-specific behavioral responses. Our method captures and fuses multi-modal signals from the built-in microphone, inertial sensor, and touchscreen without requiring users to touch a specific sensor area or hold the phone at a fixed angle or distance, offering seamless and unobtrusive authentication. Our system integrates a hybrid deep-learning model combining classification models for each modality to classify users based on these subtle patterns. Our system achieved up to 94% accuracy integrating multi-vibration stimuli and a 96% multi-modal authentication accuracy with 16 participants.

Drinking monitoring is crucial for preventing dehydration, malnutrition, and obesity. Traditional methods are prone to large errors due to inaccurate memory recall and rely heavily on active user participation. Recent studies have explored more automated approaches to reduce human effort, but these either require dedicated hardware or are restricted to laboratory settings. This work proposes an unobtrusive drinking monitoring system that can be deployed on off-the-shelf smart glasses to support real-life, daily monitoring. Specifically, we use multi-modal sensing to collect data from the smart glasses’ microphones, inertial sensors, and cameras, which contain drink-related details such as drinking sounds, jaw movements, and head vibrations. We then develop a Hybrid AI model to learn features from cross-domain sensor data and gain insights into when (timing), what (type), and how much (volume/calories) the user drinks. Experiments show that our system detects the start of a drinking activity with a median error of 0.98 seconds, recognizes seven drink types with 97.3% accuracy, and estimates drink volume and caloric intake with a 9.3% error. The proposed system is easy for new users to use and supports fine-grained drink assessment, such as distinguishing between natural and artificial sugars and estimating sugar consumption. Moreover, the system can be further integrated with large language model (LLM) analysis to generate personalized health reports, including potential health concerns such as tooth decay, nutritional assessments related to sugar consumption, and more.

Bluetooth headphones are increasingly commonly used in daily life, offering convenience and enhanced audio experience. However, these devices remain underexplored for human-beneficial applications such as serving as authenticator due to the challenges of implementing acoustic sensing on them. Particularly, Bluetooth headphones are limited by low-frequency audio bandwidth and have built-in echo cancellation algorithms, which makes the recorded signal incomplete and cannot be used for acoustic analysis. This work addresses these challenges and achieves acoustic sensing on bone-conduction Bluetooth headphones to extract a unique bone-conducted head biometric for user authentication. Specifically, the proposed system emits a user-friendly signal consisting of a welcome tone followed by a short human voice, and analyzes the received signals using a convolutional neural network developed with residual blocks to derive stable biometrics to verify users. Extensive experiments show that the proposed system can verify users’ identities with an average accuracy of 97.51% and can successfully reject 100% of replay attacks, even when an adversary eavesdrops on the authentication sound and the acoustic biometric data.

Tele-operated robots rely on real-time user behavior mapping for remote tasks, but ensuring secure authentication remains a challenge. Traditional methods, such as passwords and static biometrics, are vulnerable to spoofing and replay attacks, particularly in high-stakes, continuous interactions. This paper presents a novel anti-spoofing and anti-replay authentication approach that leverages distinctive user behavioral features extracted from haptic feedback during human–robot interactions. To evaluate our authentication approach, we collected a time-series force feedback dataset from 15 participants performing seven distinct tasks. We then developed a transformer-based deep learning model to extract temporal features from the haptic signals. By analyzing user-specific force dynamics, our method achieves over 90% accuracy in both user identification and task classification, demonstrating its potential for enhancing access control and identity assurance in tele-robotic systems.

Since the outbreak of the COVID-19 pandemic, video conferencing apps have been more broadly used to connect geographically distant people for work, school, and social interactions. These apps simulate “in-person” meetings with streamed audio and provide users with full control of their privacy. For instance, users can conveniently disable their microphones whenever they feel the need for privacy following common senses: 1) Audio signals containing semantic or contextual information pose privacy concerns; 2) Microphones are relevant only to acoustic privacy; 3) Meeting participants cannot actively intrude on each other's privacy but only opportunistically exploit accidental privacy leakages or mistakes. This paper investigates the privacy leakages that defy these assumptions. We find that any meeting participant can actively and covertly probe others' location privacy even when the webcam is disabled or virtual backgrounds are used to hide locations. More specifically, the legitimate two-way audio channel of video conferencing facilitates remote acoustic sensing, allowing an attacker to probe the users' physical surroundings and receive location-specific echo signals. However, all video conferencing systems utilize echo cancellation functions to prevent audio feedback, which inherently stops active sensing. To address this challenge, we develop a transformer-based algorithm and leverage the encoders of generative AI to counteract echo cancellation and extract stable location embeddings from severely distorted echo sounds. Furthermore, we propose two types of active acoustic sensing attacks: the in-channel echo attack, which breaks through echo cancellation by using carefully crafted signals, and the off-channel echo attack, which exploits third-party media sounds (e.g., email notification tones) to evade cancellation. We test these attacks on commercial video conferencing apps, such as Zoom, Teams, and Skype. When using only a single probing sound, our methods achieve 88.3% accuracy in recognizing recurrent places and 88.5% accuracy in identifying the contexts of new (unseen or untagged) places.

This work proposes a low-effort user authentication system for handheld devices based on active acoustic sensing. Rather than using dedicated acoustic signals, we find common media sounds like music can serve as a sensing signal to verify the phone user’s hand. Specifically, when a notification comes, the smartphone can unobtrusively verify who is holding the device and then decide whether to hide or display the sensitive notification content. Since sound and vibration co-exist, we capture two novel responses via the device’s microphone and accelerometer to describe how the individual’s contacting palm interferes with the two-domain signals, which are then described as time-frequency images and fed into a convolutional neural network-based algorithm for user authentication. Moreover, we develop a cross-domain method to validate the hard-toforge physical relationships among the smartphone’s microphone, speaker, and accelerometer, which are embedded on the same motherboard. This prevents external sounds from cheating the system. Additionally, we consider vibration alerts as a special type of musical sound and extend our method to work with the smartphone’s silent mode. Extensive experiments with ten musical sounds and five phone models show that our method verifies users with 94.5% accuracy and effectively prevents acoustic replay attacks and physical hand forgeries.

Biometrics have been widely applied for user authentication. However, existing biometric authentications are vulnerable to biometric spoofing, because they can be observed and forged. In addition, they rely on verifying biometric features that rarely change. To address this issue, we propose to verify the handgrip biometric that can be unobtrusively extracted by acoustic signals when the user holds the phone. This biometric is uniquely associated with the user’s hand geometry, body-fat ratio, and gripping strength, which are hard to reproduce. Furthermore, we propose two biometric encoding techniques (i.e., temporal-frequential and spatial) to convert static biometrics into dynamic biometric features to prevent data reuse. In particular, we develop a biometric authentication system to work with the challenge-response protocol. We encode the ultrasonic signal according to a random challenge sequence and extract a distinct biometric code as the response. We further develop two decoding algorithms to decode the biometric code for user authentication. Additionally, we investigate multiple new attacks and explore using a latent diffusion model to solve the acoustic noise discrepancies between the training and testing data to improve system performance. Extensive experiments show our system achieves 97% accuracy in distinguishing users and rejects 100% replay attacks with$ 0.6 \, s$challenge sequence.

Because of the great convenience and being not readable to humans, Quick Response (QR) codes are increasingly being utilized to offer a variety of security applications to mobile users, such as online payments, website logins, and private data sharing. To facilitate these security applications, QR codes usually contain sensitive information, such as bank account details, credit card numbers, and personal/organizational/device data, or they are specifically designed to work with cloud servers to provide security services. However, there is currently no existing solution to verify the identity of the smartphone user who scans a QR code from a Kiosk or another phone's screen. Verifying the scanner's identity is essential to ensure that financial transactions go to the correct recipient and that sensitive data is securely shared to its intended destination. This work aims to equip QR code providers with the ability to verify human scanners' identities, facilitating authorization and auditing. When a phone is held close to scan a QR code, we utilize the front camera of the code provider (a Kiosk or phone) to simultaneously verify the scanner's hand. Instead of requiring the scanner to present a stretched palm to obtain traditional hand geometries, we find that the geometry of an individual's hand, when it grips a phone, is also identifiable. We thus design a vision-based approach to extract gripping hand biometrics. We leverage the QR code's screen to cast light onto the scanner's gripping hand, ensuring adequate illumination even in low-light conditions. We then use a hand tracking tool, MediaPipe, to detect and localize the hand and develop a transformer-based algorithm to verify four types of gripping hand biometric features extracted from the hand image, including hand contour, skeleton, color, and surface. We further capture the subtle hand joint movements for liveness validation, because the user needs to click touchscreen buttons to start QR code scanning. Extensive experiments, including a long-term study spanning over 32 months, show that the system achieves 98.3% accuracy in verifying the user and mitigating 2D and 3D replay attacks. Compared to the widely used facial recognition, this approach addresses the recent struggles of identifying faces behind masks and the public concerns about privacy erosion.

Behavioral biometrics has emerged as an important security factor for user authentication. Compared to static biometrics (e.g., faces, irises, and fingerprints), using human motion behaviors for authentication causes lower concern about privacy abuse, and behavior biometrics are shown hard to be replicated by humans. In-air 3D signature is one representative of behavioral biometrics. Specifically, a user’s hand movements can be tracked by visual or wireless sensors for contact-free signature authentication, where both the fingertip trajectory and the dynamic motion features are verified to provide enhanced security. However, with the advancement of 3D printing and robot technology, we find that 1) existing hand-tracking interfaces (e.g., Leap Motion and Google MediaPipe) are easily tricked by a fake hand, and 2) a robotic arm can reproduce a user’s in-air 3D signature with high similarity regarding both trajectory and motion behaviors. Thus, this work investigates the security of in-air signatures under robot-level replays and proposes to extend the signature verification from a single-point fingertip to multiple hand joints for enhanced security. We develop the hand skeleton-based 3D signature verification system, which can be deployed on any single camera devices (2D or 3D). The key insight is that current robots could hardly replicate the minute and unique inter-joint motions of a user. In particular, we track the hand skeleton using a single camera and reconstruct/draw the trajectories of its joints in a virtual 3D space, using the color gradients to represent time-lapse and using varying line widths to describe joint significance. Based on that, we extract the three-view skeleton signatures and inter-joint motion features and develop a convolutional neural network for verification. Extensive experiments show that our system not only achieves high authentication performance but also effectively mitigates robot-level replay attacks.

While Virtual Reality (VR) applications are becoming increasingly common, efficiently verifying a VR device user before granting personal access is still a challenge. Existing VR authentication methods require users to enter PINs or draw graphical passwords using controllers. Though the entry is in the virtual space, it can be observed by others in proximity and is subject to critical security issues. Furthermore, the in-air hand movements or handheld controller-based authentications require active user participation and are not time-efficient. This work proposes a low-effort VR device authentication system based on the unique skull-reverberated sounds, which can be acquired when the user wears the VR device. Specifically, when the user puts on the VR device or is wearing it to log into an online account, the proposed system actively emits an ultrasonic signal to initiate the authentication session. The signal returning to the VR device’s microphone has been reverberated by the user’s head, which is unique in size, skull shape and mass. We thus extract head biometric information from the received signal for unobtrusive VR device authentication.Though active acoustic sensing has been broadly used on mobile devices, no prior work has ever successfully applied such techniques to commodity VR devices. Because VR devices are designed to provide users with virtual reality immersion, the echo sounds used for active sensing are unwanted and severely suppressed. The raw audio before this process is also not accessible without kernel/hardware modifications. Thus, our work further solves the challenge of active acoustic sensing under echo cancellation to enable deploying our system on off-the-shelf VR devices. Additionally, we show that the echo cancellation mechanism is naturally good to prevent acoustic replay attacks. The proposed system is developed based on an autoencoder and a convolutional neural network for biometric data extraction and recognition. Experiments with a standalone and a mobile phone VR headset show that our system efficiently verifies a user and is also replay-resistant.

Biometrics have been widely applied as personally identifiable data for user authentication. However, existing biometric authentications are vulnerable to biometric spoofing. One reason is that they are easily observable and vulnerable to physical forgeries. Examples are the apparent surface patterns of human bodies, such as fingerprints and faces. A more significant issue is that existing authentication methods are entirely built upon biometric features, which almost never change and could be obtained or learned by an adversary such as human voices. To address this inherent security issue of biometric authentications, we propose a novel acoustically extracted hand-grip biometric, which is associated with every user’s hand geometry, body-fat ratio, and gripping strength; It is implicit and available whenever they grip a handheld device. Furthermore, we integrate a coding technique in the biometric acquisition process, which encodes static biometrics into dynamic biometric features to prevent data reuse. Additionally, this low-cost method can be deployed on any handheld device that has a speaker and a microphone. In particular, we develop a challenge-response biometric authentication system, which consists of a pair of biometric encoder and decoder. We encode the ultrasonic signal according to a challenge sequence and extract a distinct biometric code as the response for each session. We then decode the biometric code to verify the user by a convolutional neural network-based algorithm, which not only examines the coding correctness but also verifies the biometric features presented by each biometric digit. Furthermore, we investigate diverse acoustic attacks to our system, by respectively assuming an adversary could present the correct code, generate similar biometric features or successfully forge both. Extensive experiments on mobile devices show that our system achieves 97% accuracy to distinguish users and rejects 100% replay and synthesis attacks with 6-digit codes.

Motion-controlled robotic arms allow a user to interact with a remote real world without physically reaching it. By connecting cyberspace to the physical world, such interactive teleoperations are promising to improve remote education, virtual social interactions, and online participatory activities. In this work, we build up a motion-controlled robotic arm framework comprising a robotic arm end and a user end, which are connected via a network and responsible for manipulator control and motion capture, respectively. To protect the system access, we propose to verify who is controlling the robotic arm by examining the robotic arm’s behavior, which adds a second security layer in addition to the system login credentials. We show that a robotic arm’s motion inherits its human controller’s behavioral biometric in interactive control scenarios. By extracting the angle readings of the robotic arm’s all joints, the proposed user authentication approach reconstructs the robotic arm’s end-effector movement trajectory that follows the user’s hand. Furthermore, we derive the unique robotic motion features to capture the user’s behavioral biometric embedded in the robot motions and develop learning-based algorithms to verify the robotic arm user to be one of the enrolled users or a nonuser. Extensive experiments show that our system achieves 94% accuracy to distinguish users while preventing user identity spoofing attacks with 95% accuracy.

This work enables mobile user authentication via finger inputs on ubiquitous surfaces leveraging low-cost physical vibration. The system we proposed extends finger-input authentication beyond touch screens to any solid surface for IoT devices (e.g., smart access systems and IoT appliances). Unlike passcode or biometrics-based solutions, it integrates passcode, behavioral and physiological characteristics, and surface dependency together to provide a low-cost, tangible and enhanced security solution. The proposed system builds upon a touch sensing technique with vibration signals that can operate on surfaces constructed from a broad range of materials. New algorithms are developed to discriminate fine-grained finger inputs and supports three independent passcode secrets including PIN number, lock pattern, and simple gestures by extracting unique features in the frequency domain to capture both behavioral and physiological characteristics including contacting area, touching force, and etc. The system is implemented using a single pair of low-cost portable vibration motor and receiver that can be easily attached to any surface (e.g., a door panel, a stovetop or an appliance). Extensive experiments demonstrate that our system can authenticate users with high accuracy (e.g., more than 97 percent within two trials), low false positive rate (e.g., less 2 percent) and is robust to various types of attacks.

Recent advances in speaker verification and speech processing technology have seen voice authentication being adopted on a wide scale in commercial applications like online banking and customer care support and on devices such as smartphones and IoT voice assistant systems. However, it has been shown that the current voice authentication systems can be ineffective against voice synthesis attacks that mimic a user's voice to high precision. In this work, we suggest a paradigm shift from the traditional voice authentication systems operating in the audio domain but susceptible to speech synthesis attacks (in the same audio domain). We leverage a motion sensor's capability to pick up phonatory vibrations, that can help to uniquely identify a user via voice signatures in the vibration domain. The user's speech is played/echoed back by a device's speaker for a short duration (hence our method is termed EchoVib) and the resulting non-linear phonatory vibrations are picked up by the motion sensor for speaker recognition. The uniqueness of the device's speaker and its accelerometer results in a device-specific fingerprint in response to the echoed speech. The use of the vibration domain and its non-linear relationship with audio allows EchoVib to resist the state-of-the-art voice synthesis attacks, shown to be successful in the audio domain.

Exercise intensity monitoring of physical activities has drawn increasingly attention as the awareness of the exercise intensity is of great importance for a person to achieve optimal training outcomes. For example, over-training could lead to excessive fatigue and loss of motivation for exercise. Traditional exercise intensity monitoring systems utilize GPS data to track the user’s intensity of cardio activities through his/her position and speed. Such systems however become invalid for indoor exercises on stationary fitness equipments such as the treadmill or exercise bike. Recent work in using body-worn sensors to track the user’s heart rate for exercise intensity monitoring usually involves additional wearable sensors which are only available on some particular fitness equipments, and thus are hard to be used in all occasions. This work presents an exercise intensity monitoring system which is capable of detecting a person’s exercise intensity via smartphones. Our system exploits the off-the-shelf smartphone and its headphone to capture the user’s breathing sound. Given the captured acoustic data, our system performs data pre-processing to remove the environmental noise and identify the non-silent acoustic frames based on the signal energy. Our system then conducts breathing event detection for non-silent frames, and further calibrates the detection results by utilizing the high correlation between breathing cycles to improve the detection accuracy. Moreover, our system can estimate the person’s exercise intensity based on features extracted from the frames which contain breathing sound. Our experiments involving 9 subjects over four-month time period demonstrate that our proposed exercise intensity monitoring system is robust and accurate in both indoor and outdoor environments.

Handheld phone distraction is the leading cause of traffic accidents. However, few efforts have been devoted to detecting when the phone distraction happens, which is a critical input for taking immediate safety measures. This work proposes a phone-use monitoring system, which detects the start of the driver’s handheld phone use and eliminates the distraction at once. Specifically, the proposed system emits periodic ultrasonic pulses to sense if the phone is being held in hand or placed on support surfaces (e.g., seat and cup holder) by capturing the unique signal interference resulted from the contact object’s damping, reflection and refraction. We derive the short-time Fourier transform from the microphone data to describe such impacts and develop a CNN-based binary classifier to discriminate the phone use between the handheld and the handsfree status. Additionally, we design an adaptive window-based filter to correct the classification errors and identify each handheld phone distraction instance, including its start, end, and duration. Extensive experiments with fourteen people, three phones and two car models show that our system achieves 99% accuracy of recognizing handheld phone-use instances and 0.76-second median error to estimate the distraction’s start time.

This work proposes a media sound-based authentication method to protect smartphone notification privacy unobtrusively, which wisely hides or presents sensitive content by verifying who is holding the phone. We show that media sounds, such as the melodies of notification tones (e.g., iPhone message and Samsung whistle) can be directly used to sense and verify the user's gripping hand. Because sounds and vibrations co-exist, we capture two novel responses via the smartphone mic and accelerometer to describe how the individual's contacting palm interferes with the signals in two different domains. Based on the two responses, we develop a convolutional neural network-based algorithm to verify the user. Moreover, because the smartphone sensors are all embedded on the same motherboard, we develop a cross-domain method to validate such hard-to-forge physical relationships among the mic, speaker and accelerometer. They prevent external sounds from cheating the system. Additionally, we consider the notification vibration as a special type of media sound, which also results in two responses, and extend our method to work in the silent mode. Extensive experiments with ten notification tones and four phone models show that our system verifies users with 95% accuracy and prevents replay sounds with 100% accuracy.

Motion-controlled robots allow a user to interact with a remote real world without physically reaching it. By connecting cyberspace to the physical world, such interactive teleoperations are promising to improve remote education, virtual social interactions and online participatory activities. This work builds up a motion-controlled robotic arm framework and proposes to verify who is controlling the robotic arm by examining the robotic arm's behavior. We show that a robotic arm's motion inherits its human controller's behavioral biometric in interactive control scenarios. Furthermore, we derive the unique robotic motion features to capture the user's behavioral biometric embedded in the robot motions and develop learning-based algorithms to verify the robotic arm user. Extensive experiments show that our system achieves high accuracy to distinguish users while using the robot's behaviors.

We present a demonstration of a human perception-enhanced camera system for web conferencing that protects the user's privacy. Given that people easily forget about their active camera during web conferences, the system advertises the camera's active status via its motions to remind users that they are being watched by others. This prevents inadvertent privacy leakage. The system is developed based on a motorized camera, which moves according to the user's head coordinates just like an eye is looking at the user's face in front of the desk rather than remotely or virtually. The basic idea is to exploit the original human body sense of environmental motions for human-camera interaction, which does not require looking straight at the camera or its LED light to actively check its status. In this demonstration, we showcase our implementation of the human perception-enhanced camera system and invite participants to use the system for web conferences (e.g., Zoom and Google Hangout), which illustrates the system's ability to extend the virtual social interaction to the physical world and the effectiveness of using the camera motion as a non-intrusive awareness indicator.

Phone usage while driving is unanimously considered a really dangerous habit due to a strong correlation with road accidents. This paper proposes a phone-use monitoring system that detects the driver's handheld phone use and eliminates the distraction at once. Specifically, the proposed system emits periodic ultrasonic pulses to sense if the phone is being held in hand or placed on support surfaces (e.g., seat and cup holder) by capturing the unique signal interference resulted from the contact object's damping, reflection and refraction. We derive the short-time Fourier transform from the microphone data to describe such impacts and develop a CNN-based binary classifier to discriminate the phone use between the handheld and the handsfree status. Additionally, we design a classification error correction filter to correct the classification errors during the monitoring. The experiments with six people, one phone and one car model show that our system achieves 99% accuracy in recognizing handheld phone-use activities.

In this paper, we build a speech privacy attack that exploits speech reverberations from a smartphone's inbuilt loudspeaker captured via a zero-permission motion sensor (accelerometer). We design our attack Spearphone, and demonstrate that speech reverberations from inbuilt loudspeakers, at an appropriate loudness, can impact the accelerometer, leaking sensitive information about the speech. In particular, we show that by exploiting the affected accelerometer readings and carefully selecting feature sets along with off-the-shelf machine learning techniques, Spearphone can perform gender classification (accuracy over 90%) and speaker identification (accuracy over 80%) for the audio/video playback on the smartphone for our recorded dataset. We use lightweight classifiers and an off-the-shelf machine learning tool so that the attacking effort is minimized, making our attack practical. Our results with testing the attack on a voice call and voice assistant response were also encouraging, showcasing the impact of the proposed attack. In addition, we perform speech recognition and speech reconstruction to extract more information about the eavesdropped speech to an extent. Our work brings to light a fundamental design vulnerability in many currently-deployed smartphones, which may put people's speech privacy at risk while using the smartphone in the loudspeaker mode during phone calls, media playback or voice assistant interactions.

Due to the open nature of voice input, voice assistant (VA) systems (e.g., Google Home and Amazon Alexa) are vulnerable to various security and privacy leakages (e.g., credit card numbers, passwords), especially when issuing critical user commands involving large purchases, critical calls, etc. Though the existing VA systems may employ voice features to identify users, they are still vulnerable to various acoustic-based attacks (e.g., impersonation, replay, and hidden command attacks). In this work, we propose a training-free voice authentication system, WearID, leveraging the cross-domain speech similarity between the audio domain and the vibration domain to provide enhanced security to the ever-growing deployment of VA systems. In particular, when a user gives a critical command, WearID exploits motion sensors on the user’s wearable device to capture the aerial speech in the vibration domain and verify it with the speech captured in the audio domain via the VA device’s microphone. Compared to existing approaches, our solution is low-effort and privacy-preserving, as it neither requires users’ active inputs (e.g., replying messages/calls) nor to store users’ privacy-sensitive voice samples for training. In addition, our solution exploits the distinct vibration sensing interface and its short sensing range to sound (e.g., 25cm) to verify voice commands. Examining the similarity of the two domains’ data is not trivial. The huge sampling rate gap (e.g., 8000Hz vs. 200Hz) between the audio and vibration domains makes it hard to compare the two domains’ data directly, and even tiny data noises could be magnified and cause authentication failures. To address the challenges, we investigate the complex relationship between the two sensing domains and develop a spectrogram-based algorithm to convert the microphone data into the lower-frequency “ motion sensor data” to facilitate cross-domain comparisons. We further develop a user authentication scheme to verify that the received voice command originates from the legitimate user based on the cross-domain speech similarity of the received voice commands. We report on extensive experiments to evaluate the WearID under various audible and inaudible attacks. The results show WearID can verify voice commands with 99.8% accuracy in the normal situation and detect 97.2% fake voice commands from various attacks, including impersonation/replay attacks and hidden voice/ultrasound attacks.

Many existing identification approaches require active user input, specialized sensing hardware, or personally identifiable information such as fingerprints or face scans. In this paper, we propose EchoLock, a low-effort identification scheme that validates the user by sensing hand geometry via commodity microphones and speakers. EchoLock can serve as a complementary verification method for high-end devices or as a stand-alone user identification scheme for lower-end devices without using privacy-sensitive features. In addition to security applications, our system can also personalize user interactions with smart devices, such as automatically adapting settings or preferences when different people are holding smart remotes. To this end, we study the impact of hands on structure borne sound propagation in mobile devices and develop a user identification scheme that can measure, quantify, and exploit distinct sound reflections in order to differentiate distinct identities. Particularly, we propose a non-intrusive hand sensing technique to derive unique acoustic features in both time and frequency domain, which can effectively capture the physiological and behavioral traits of a user's hand (e.g., hand contours, finger sizes, holding strengths, and holding styles). Furthermore, learning-based algorithms are developed to robustly identify the user under various environments and conditions. We conduct extensive experiments with 20 participants, gathering 80,000 hand geometry samples using different hardware setups across 160 key use case scenarios. Our results show that EchoLock is capable of identifying users with over 94% accuracy, without requiring any active user input.

Eating well plays a key role in people's overall health and wellbeing. Studies have shown that many health-related problems such as obesity, diabetes and anemia are closely associated with people's unhealthy eating habits (e.g., skipping meals, eating irregularly and overeating). Thus, keeping track of diet is becoming more important. Traditional eating monitoring solutions relying on self-report remain an onerous task, while the recent trends requiring users to wear dedicated yet expensive hardware are cumbersome. To overcome these limitations, in this paper, we develop a device-free eating monitoring system using WiFi-enabled devices (e.g., smartphone or laptop). Our system aims to automatically monitor users' eating activities by identifying the fine-grained eating motions and detecting the minute movements during chewing and swallowing. In particular, our system distinguishes eating from non-eating activities by using K-means clustering with principal component analysis on the extracted Channel State Information (CSI) from WiFi signals. It further adopts a soft decision-based eating motion classification through identifying the utensils (e.g., using a folk, knife, spoon or bare hands) in use. Moreover, we propose a minute motion reconstruction method to identify chewing and swallowing through detecting users' minute facial muscle movements. The derived fine-grained eating monitoring results are beneficial to the understanding of users' eating behaviors and estimation of food intake types and amounts. Extensive experiments with 20 users over 1600-minute eating show that the proposed system can recognize the user's eating motions with up to 95% accuracy and estimate the chewing and swallowing amount within 10% percentage error.

As the most common personal devices, smartphones contain the user's private information. While people use mobile devices anytime and anywhere, the sensitive contents might be leaked from the screens. The smartphone notifications cause such privacy leakages even on a lock screen. With the aim to alert the user of an event (e.g., text messages, phone calls and calendar reminders), these onscreen notifications usually contain the sender's name and even a clip of the contents for preview. Such information, if not displayed appropriately, may cause the leakages of the user's social relations, personal hobbies and private message contents. This work focuses on wisely displaying the notifications to avoid leaking the user's privacy. We develop an unobtrusive user authentication system to confirm the user identity via their gripping-hands before displaying notifications. In particular, we carefully design an inaudible acoustic signal and emit it from the smartphone speaker to sense the gripping hand, when there is a need to push notifications. The signal propagating to the smartphone's microphones carries the user's biometric information related to the gripping hand (e.g., palm size and gripping strength). We further derive the Mel Frequency Cepstral Coefficient time series and develop a machine learning-based algorithm to identify the user. The experimental results show that our system can identify 8 users with 92% accuracy.

The explosive usage of mobile devices enables conducting electronic transactions involving direct signature on such devices. Thus, user signature verification becomes critical to ensure the success deployment of online transactions such as approving legal documents and authenticating financial transactions. Existing approaches mainly focus on user verification targeting the unlocking of mobile devices or performing continuous verification based on a user's behavioral traits. Few studies provide efficient real-time user signature verification. In this work, we propose a critical segment based online signature verification system to secure mobile transactions on multi-touch mobile devices. Our system identifies and exploits the segments which remain invariant within a user's signature to capture the intrinsic signing behavior embedded in each user's signature. Our system extracts useful features from a user's signature that describe both the geometric layout of the signature as well as behavioral and physiological characteristics in the user's signing process. Given the input signatures for user enrollment, our system further designs a quality score to identify the problematic signature sets to achieve robust user signature profile construction. Moreover, we develop the signature normalization and interpolation methods to achieve robust signature verification in the presence of signature geometric distortions caused by different writing sizes, orientations and locations on touch screens. Our experimental evaluation of 25 subjects over six months time period shows that our system is highly accurate in provide signature verification and robust to signature forging attacks.

Voice access technologies are widely adopted in mobile devices and voice assistant systems as a convenient way of user interaction. Recent studies have demonstrated a potentially serious vulnerability of the existing voice interfaces on these systems to "hidden voice commands". This attack uses synthetically rendered adversarial sounds embedded within a voice command to trick the speech recognition process into executing malicious commands, without being noticed by legitimate users.

Mobile payment has drawn considerable attention due to its convenience of paying via personal mobile devices at anytime and anywhere, and passcodes (i.e., PINs or patterns) are the first choice of most consumers to authorize the payment. This paper demonstrates a serious security breach and aims to raise the awareness of the public that the passcodes for authorizing transactions in mobile payments can be leaked by exploiting the embedded sensors in wearable devices (e.g., smartwatches). We present a passcode inference system, WristSpy, which examines to what extent the user's PIN/pattern during the mobile payment could be revealed from a single wrist-worn wearable device under different passcode input scenarios involving either two hands or a single hand. In particular, WristSpy has the capability to accurately reconstruct fine-grained hand movement trajectories and infer PINs/patterns when mobile and wearable devices are on two hands through building a Euclidean distance-based model and developing a training-free parallel PIN/pattern inference algorithm. When both devices are on the same single hand, a highly challenging case, WristSpy extracts multi-dimensional features by capturing the dynamics of minute hand vibrations and performs machine-learning based classification to identify PIN entries. Extensive experiments with 15 volunteers and 1600 passcode inputs demonstrate that an adversary is able to recover a user's PIN/pattern with up to 92% success rate within 5 tries under various input scenarios.

Sleep monitoring has drawn increasing attention as sleep quality is important to maintain a person's well-being. For instance, serious health problems, such as cardiovascular disease, fatigue, or depression, are usually associated with inadequate and irregular sleep. Traditional sleep monitoring systems involve wearable sensors with professional installation, and thus are usually limited to clinical usage. Recent work for sleep monitoring can detect several sleep events, such as coughing and snoring, using smartphone sensors. However, such coarse-grained sleep monitoring is unable to detect the breathing rate which is an important health indicator. In this paper, we present a fine-grained sleep monitoring system to detect the breathing rate and sleep events simultaneously by leveraging smartphones. Our system exploits the readily available smartphone earphone placed close to the user to reliably capture the human breathing sound. Given the captured acoustic sound, noise reduction is performed to remove the environmental noise and the breathing rate is then identified based on the signal envelope detection. Our system can further detect some sleep events, including snoring, coughing, turning over, and getting up, based on the features extracted from the acoustic sound. Moreover, we develop a body movement-assisted sleep event detection method to provide higher detection accuracy by further exploiting the user's body movement patterns captured by the accelerometer embedded on smartphones. Our extensive experiments involving nine subjects over six months confirm the effectiveness of our proposed system on breathing rate monitoring and sleep events detection under various environments. By combining breathing rate and sleep events, our system can provide noninvasive and continuous fine-grained sleep monitoring for healthcare related applications, such as sleep apnea monitoring, as evidenced by our experimental study.

Mobile payment has drawn considerable attention due to its convenience of paying via personal mobile devices at anytime and anywhere, and passcodes (i.e., PINs) are the first choice of most consumers to authorize the payment. This work demonstrates a serious security breach and aims to raise the awareness of the public that the passcodes for authorizing transactions in mobile payments can be leaked by exploiting the embedded sensors in wearable devices (e.g., smartwatches). We present a passcode inference system, which examines to what extent the user's PIN during mobile payment could be revealed from a single wrist-worn wearable device under different input scenarios involving either two hands or a single hand. Extensive experiments with 15 volunteers demonstrate that an adversary is able to recover a user's PIN with high success rate within 5 tries under various input scenarios.

The proliferation of wearable devices, e.g., smartwatches and activity trackers, with embedded sensors has already shown its great potential on monitoring and inferring human daily activities. This paper reveals a serious security breach of wearable devices in the context of divulging secret information (i.e., key entries) while people are accessing key-based security systems. Existing methods of obtaining such secret information rely on installations of dedicated hardware (e.g., video camera or fake keypad), or training with labeled data from body sensors, which restrict use cases in practical adversary scenarios. In this work, we show that a wearable device can be exploited to discriminate mm-level distances and directions of the user's fine-grained hand movements, which enable attackers to reproduce the trajectories of the user's hand and further to recover the secret key entries. In particular, our system confirms the possibility of using embedded sensors in wearable devices, i.e., accelerometers, gyroscopes, and magnetometers, to derive the moving distance of the user's hand between consecutive key entries regardless of the pose of the hand. Our Backward PIN-Sequence Inference algorithm exploits the inherent physical constraints between key entries to infer the complete user key entry sequence. Extensive experiments are conducted with over 7,000 key entry traces collected from 20 adults for key-based security systems (i.e., ATM keypads and regular keyboards) through testing on different kinds of wearables. Results demonstrate that such a technique can achieve 80 percent accuracy with only one try and more than 90 percent accuracy with three tries. Moreover, the performance of our system is consistently good even under low sampling rate and when inferring long PIN sequences. To the best of our knowledge, this is the first technique that reveals personal PINs leveraging wearable devices without the need for labeled training data and contextual information.

The goal of this work is to enable user authentication via finger inputs on ubiquitous surfaces leveraging low-cost physical vibration. We propose VibWrite that extends finger-input authentication beyond touch screens to any solid surface for smart access systems (e.g., access to apartments, vehicles or smart appliances). It integrates passcode, behavioral and physiological characteristics, and surface dependency together to provide a low-cost, tangible and enhanced security solution. VibWrite builds upon a touch sensing technique with vibration signals that can operate on surfaces constructed from a broad range of materials. It is significantly different from traditional password-based approaches, which only authenticate the password itself rather than the legitimate user, and the behavioral biometrics-based solutions, which usually involve specific or expensive hardware (e.g., touch screen or fingerprint reader), incurring privacy concerns and suffering from smudge attacks. VibWrite is based on new algorithms to discriminate fine-grained finger inputs and supports three independent passcode secrets including PIN number, lock pattern, and simple gestures by extracting unique features in the frequency domain to capture both behavioral and physiological characteristics such as contacting area, touching force, and etc. VibWrite is implemented using a single pair of low-cost vibration motor and receiver that can be easily attached to any surface (e.g., a door panel, a desk or an appliance). Our extensive experiments demonstrate that VibWrite can authenticate users with high accuracy (e.g., over 95% within two trials), low false positive rate (e.g., less 3%) and is robust to various types of attacks.

While the mobile users enjoy the anytime anywhere Internet access by connecting their mobile devices through Wi-Fi services, the increasing deployment of access points (APs) have raised a number of privacy concerns. This paper explores the potential of smartphone privacy leakage caused by surrounding APs. In particular, we study to what extent the users' personal information such as social relationships and demographics could be revealed leveraging simple signal information from APs without examining the Wi-Fi traffic. Our approach utilizes users' activities at daily visited places derived from the surrounding APs to infer users' social interactions and individual behaviors. Furthermore, we develop two new mechanisms: the Closeness-based Social Relationships Inference algorithm captures how closely people interact with each other by evaluating their physical closeness and derives fine-grained social relationships, whereas the Behavior-based Demographics Inference method differentiates various individual behaviors via the extracted activity features (e.g., activeness and time slots) at each daily place to reveal users' demographics. Extensive experiments conducted with 21 participants' real daily life including 257 different places in three cities over a 6-month period demonstrate that the simple signal information from surrounding APs have a high potential to reveal people's social relationships and infer demographics with an over 90% accuracy when using our approach.

Rogue access point (AP) has emerged as an important security problem in WLANs. However, it is a challenge task to localize the rogue AP with both high accuracy and minimal infrastructure cost. Either expensive professional infrastructure (e.g., multiple wireless sniffers) or additional hardware (e.g., directional antenna) need to be pre-deployed for rogue AP localization with high cost. Moreover, existing methods using Received Signal Strength (RSS) result in a large error as RSS is suffered from the multipath and shadowing effects in complex wireless environment. In this work, we exploit the channel state information (CSI), which is readily available from commercial Wi-Fi devices, to locate the rogue AP with high accuracy. We use only a single off-the-shelf Wi-Fi device for rogue AP localization which involves minimal infrastructure requirement. Our proposed rogue AP localization framework consists of two components: direction determination and position estimation. The direction determination can be carried out by using the human blocking effect on the CSI amplitude or phase. The multiple antennas on the Wi-Fi devices can be further utilized to enhance the rogue AP direction estimation. Given the estimated direction, two schemes are proposed to pinpoint the position of the rogue AP: determining directions at multiple locations grounded on triangulation and walking towards the rogue AP with direction adjustment. Results from extensive experiments in both indoor and outdoor environments show that our framework can achieve more practical and accurate rogue AP localization when comparing with the existing RSS-based approach.

The proliferation of wearable devices, e.g., smartwatches and activity trackers, with embedded sensors has already shown its great potential on monitoring and inferring human daily activities. This paper reveals a serious security breach of wearable devices in the context of divulging secret information (i.e., key entries) while people accessing key-based security systems. Existing methods of obtaining such secret information relies on installations of dedicated hardware (e.g., video camera or fake keypad), or training with labeled data from body sensors, which restrict use cases in practical adversary scenarios. In this work, we show that a wearable device can be exploited to discriminate mm-level distances and directions of the user's fine-grained hand movements, which enable attackers to reproduce the trajectories of the user's hand and further to recover the secret key entries. In particular, our system confirms the possibility of using embedded sensors in wearable devices, i.e., accelerometers, gyroscopes, and magnetometers, to derive the moving distance of the user's hand between consecutive key entries regardless of the pose of the hand. Our Backward PIN-Sequence Inference algorithm exploits the inherent physical constraints between key entries to infer the complete user key entry sequence. Extensive experiments are conducted with over 5000 key entry traces collected from 20 adults for key-based security systems (i.e. ATM keypads and regular keyboards) through testing on different kinds of wearables. Results demonstrate that such a technique can achieve 80% accuracy with only one try and more than 90% accuracy with three tries, which to our knowledge, is the first technique that reveals personal PINs leveraging wearable devices without the need for labeled training data and contextual information.

In this work, we propose the first PIN number based authentication system, which can be deployed on ubiquitous surfaces, leveraging physical vibration signals. The proposed system aims to integrate PIN number, behavioral and physiological characteristics together to provide enhanced security. Different from the existing password-based approaches, the proposed system builds upon a touch sensing technique using vibration signals that can operate on any solid surface. In this poster, we explore the feasibility of using vibration signals for ubiquitous user authentication and develop algorithms that identify fine-grained finger inputs with different password secrets (e.g., PIN sequences). We build a prototype using a vibration transceiver that can be attached to any surface (e.g., a door or a desk) easily. Our experiments in office environments with multiple users demonstrate that we can achieve high authentication accuracy with a low false negative rate.

Sleep monitoring has drawn increasingly attention as the quality and quantity of the sleep are important to maintain a person's health and well-being. For example, inadequate and irregular sleep are usually associated with serious health problems such as fatigue, depression and cardiovascular disease. Traditional sleep monitoring systems, such as PSG, involve wearable sensors with professional installations, and thus are limited to clinical usage. Recent work in using smartphone sensors for sleep monitoring can detect several events related to sleep, such as body movement, cough and snore. Such coarse-grained sleep monitoring however is unable to detect the breathing rate which is an important vital sign and health indicator. This work presents a fine-grained sleep monitoring system which is capable of detecting the breathing rate by leveraging smartphones. Our system exploits the readily available smartphone earphone placed close to the user to reliably capture the human breathing sound. Given the captured acoustic sound, our system performs noise reduction to remove environmental noise and then identifies the breathing rate based on the signal envelope detection. Our system can further detect detailed sleep events including snore, cough, turn over and get up based on the acoustic features extracted from the acoustic sound. Our experimental evaluation of six subjects over six months time period demonstrates that the breathing rate monitoring and sleep events detection are highly accurate and robust under various environments. By combining breathing rate and sleep events, our system can provide continuous and noninvasive fine-grained sleep monitoring for healthcare related applications, such as sleep apnea monitoring as evidenced by our experimental study.

Sleep monitoring has drawn increasingly attention as the quality and quantity of the sleep are important for maintaining a person's health and well-being. For example, inadequate and irregular sleep are usually associated with serious health problems such as fatigue, depression and cardiovascular disease. Traditional sleep monitoring systems, such as PSG, involve wearable sensors with professional installations, and thus are limited to clinical usage. Recent work in using smartphone sensors for sleep monitoring can detect several events related to sleep, such as body movement, cough and snore. Such coarse-grained sleep monitoring however is unable to detect the breathing rate which is a vital sign and health indicator. This work presents a fine-grained sleep monitoring system which is capable of detecting the breathing rate by leveraging smartphones. Our system exploits the readily available smartphone earphone that placed close to the user to capture the breath sound reliably. Given the captured acoustic signal, our system performs noise reduction to remove environmental noise and then identifies the breathing rate based on the signal envelope detection. Our experimental evaluation of six subjects over six months time period demonstrates that the breathing rate monitoring is highly accurate and robust under various environments. This strongly indicates the feasibility of using the smartphone and its earphone to perform continuous and noninvasive fine-grained sleep monitoring.

Many studies have presented the order sense results of information transmission capacity and packet delivery delay in mobile ad hoc networks (MANETs). To achieve the fundamental understanding of MANETs, we focus on deriving the closed-form expressions of the network capacity and end-to-end delay. A MANET with the generalized correlated mobility model is considered in this paper, where the mobility of nodes clustered in one group is confined within a specified area, and multiple groups move uniformly across the network. We also leverage limited packet redundancy to speed up the packet transmission, i.e., each source node is allowed to distribute at most f copies of each packet in its delivery process. Specifically, we first propose an effective multi-hop scheduling-routing scheme under the correlated mobility model, and then develop the closed-form expressions of both per node throughput capacity and expected end-to-end delay. We further explore the tradeoff between throughput capacity and packet delay by using packet redundancy f. The simulation studies validate our theoretical results.

The two hop relay algorithms with redundancy are attractive for mobile ad hoc networks (MANET) since they are simple and efficient. In this paper, we extend the analysis of the f-cast two-hop relay algorithm under i.i.d. mobility model to the case of corrected nodes movements, where the source node is allowed to send up to f copies of a packet and the clustered nodes move uniformly across the network. We first provide an effective scheduling-routing algorithm for packet relay inter- and intra-cluster and then explore the scaling laws of throughput capacity of the considered network. This result helps us to study the impact of both the packet redundancy and correlated node movement, and guide us to find the maximum possible throughput capacity through a proper setting of redundancy f.