Unlike static biometrics (e.g., faces and fingerprints), dynamic behavioral biometrics are believed to be more difficult to replicate. This paper investigates the security of behavioral biometrics considering the advancements in robotics and AI, particularly as humanoid robots, like Tesla Optimus, are expected to be mass-produced in the coming years. We find that general robotic arms have already gained the capability to reproduce human hand motion trajectories. However, using robots to replicate a user’s behavioral biometrics for attacks remains under-explored due to two long-standing challenges: 1) how to obtain the user’s complex behavioral biometrics through practical eavesdropping (not just trajectories); 2) how to replicate the user’s behavioral kinematics based on the eavesdropped data using a real robot. This work is the first to comprehensively address the two challenges. We develop the point-wise GAN-based Robot Replay Attack (GANRRA) to demonstrate a practical human behavioral replay attack using a hidden camera and a physical robot. GANRRA utilizes a hidden camera to eavesdrop on the user’s hand motions and employs a generative adversarial network to reconstruct the motion data, addressing the sensor discrepancies between the legitimate sensor and the hidden camera and maximizing the behavioral feature similarities. The reconstructed motion data is converted into velocity commands for a robot to execute point by point, replicating both hand movement trajectories and behavioral biometric features. For experiments, we implement an in-air signature system using two existing hand-tracking systems and fool them using a robotic arm attached with a fake hand. Results show that GANRRA reproduces in-air signatures with a 73.1% success rate. To address such robot-relay threats, a novel defense mechanism based on multi-joint behaviors is proposed.

While keyboard typing has been the most common way of inputting texts, handwriting still plays an important role in generating, inputting, or recording information like filling out essential/private forms. Considerable research has been done to identify and demonstrate the risk of keystroke-inference attacks. However, little has been done on handwriting inference despite its high risk of leaking sensitive information. To assess this under-explored risk of information leakage, we present a novel handwriting-inference attack, called HW-Spy, by tracing the victim's pen-tail movements when both the pen tip and the writing surface are outside the view of the attacker's camera, which usually happens when the victim is multitasking during an online meeting, when the victim's writing scene (in a public space) is recorded by a remote camera, or when the victim's writing behaviors are captured by the surveillance camera in a bank/dealership/realty office.

Since the outbreak of the COVID-19 pandemic, video conferencing apps have been more broadly used to connect geographically distant people for work, school, and social interactions. These apps simulate “in-person” meetings with streamed audio and provide users with full control of their privacy. For instance, users can conveniently disable their microphones whenever they feel the need for privacy following common senses: 1) Audio signals containing semantic or contextual information pose privacy concerns; 2) Microphones are relevant only to acoustic privacy; 3) Meeting participants cannot actively intrude on each other's privacy but only opportunistically exploit accidental privacy leakages or mistakes. This paper investigates the privacy leakages that defy these assumptions. We find that any meeting participant can actively and covertly probe others' location privacy even when the webcam is disabled or virtual backgrounds are used to hide locations. More specifically, the legitimate two-way audio channel of video conferencing facilitates remote acoustic sensing, allowing an attacker to probe the users' physical surroundings and receive location-specific echo signals. However, all video conferencing systems utilize echo cancellation functions to prevent audio feedback, which inherently stops active sensing. To address this challenge, we develop a transformer-based algorithm and leverage the encoders of generative AI to counteract echo cancellation and extract stable location embeddings from severely distorted echo sounds. Furthermore, we propose two types of active acoustic sensing attacks: the in-channel echo attack, which breaks through echo cancellation by using carefully crafted signals, and the off-channel echo attack, which exploits third-party media sounds (e.g., email notification tones) to evade cancellation. We test these attacks on commercial video conferencing apps, such as Zoom, Teams, and Skype. When using only a single probing sound, our methods achieve 88.3% accuracy in recognizing recurrent places and 88.5% accuracy in identifying the contexts of new (unseen or untagged) places.

This work proposes a low-effort user authentication system for handheld devices based on active acoustic sensing. Rather than using dedicated acoustic signals, we find common media sounds like music can serve as a sensing signal to verify the phone user’s hand. Specifically, when a notification comes, the smartphone can unobtrusively verify who is holding the device and then decide whether to hide or display the sensitive notification content. Since sound and vibration co-exist, we capture two novel responses via the device’s microphone and accelerometer to describe how the individual’s contacting palm interferes with the two-domain signals, which are then described as time-frequency images and fed into a convolutional neural network-based algorithm for user authentication. Moreover, we develop a cross-domain method to validate the hard-toforge physical relationships among the smartphone’s microphone, speaker, and accelerometer, which are embedded on the same motherboard. This prevents external sounds from cheating the system. Additionally, we consider vibration alerts as a special type of musical sound and extend our method to work with the smartphone’s silent mode. Extensive experiments with ten musical sounds and five phone models show that our method verifies users with 94.5% accuracy and effectively prevents acoustic replay attacks and physical hand forgeries.

Biometrics have been widely applied for user authentication. However, existing biometric authentications are vulnerable to biometric spoofing, because they can be observed and forged. In addition, they rely on verifying biometric features that rarely change. To address this issue, we propose to verify the handgrip biometric that can be unobtrusively extracted by acoustic signals when the user holds the phone. This biometric is uniquely associated with the user’s hand geometry, body-fat ratio, and gripping strength, which are hard to reproduce. Furthermore, we propose two biometric encoding techniques (i.e., temporal-frequential and spatial) to convert static biometrics into dynamic biometric features to prevent data reuse. In particular, we develop a biometric authentication system to work with the challenge-response protocol. We encode the ultrasonic signal according to a random challenge sequence and extract a distinct biometric code as the response. We further develop two decoding algorithms to decode the biometric code for user authentication. Additionally, we investigate multiple new attacks and explore using a latent diffusion model to solve the acoustic noise discrepancies between the training and testing data to improve system performance. Extensive experiments show our system achieves 97% accuracy in distinguishing users and rejects 100% replay attacks with$ 0.6 \, s$challenge sequence.

Because of the great convenience and being not readable to humans, Quick Response (QR) codes are increasingly being utilized to offer a variety of security applications to mobile users, such as online payments, website logins, and private data sharing. To facilitate these security applications, QR codes usually contain sensitive information, such as bank account details, credit card numbers, and personal/organizational/device data, or they are specifically designed to work with cloud servers to provide security services. However, there is currently no existing solution to verify the identity of the smartphone user who scans a QR code from a Kiosk or another phone's screen. Verifying the scanner's identity is essential to ensure that financial transactions go to the correct recipient and that sensitive data is securely shared to its intended destination. This work aims to equip QR code providers with the ability to verify human scanners' identities, facilitating authorization and auditing. When a phone is held close to scan a QR code, we utilize the front camera of the code provider (a Kiosk or phone) to simultaneously verify the scanner's hand. Instead of requiring the scanner to present a stretched palm to obtain traditional hand geometries, we find that the geometry of an individual's hand, when it grips a phone, is also identifiable. We thus design a vision-based approach to extract gripping hand biometrics. We leverage the QR code's screen to cast light onto the scanner's gripping hand, ensuring adequate illumination even in low-light conditions. We then use a hand tracking tool, MediaPipe, to detect and localize the hand and develop a transformer-based algorithm to verify four types of gripping hand biometric features extracted from the hand image, including hand contour, skeleton, color, and surface. We further capture the subtle hand joint movements for liveness validation, because the user needs to click touchscreen buttons to start QR code scanning. Extensive experiments, including a long-term study spanning over 32 months, show that the system achieves 98.3% accuracy in verifying the user and mitigating 2D and 3D replay attacks. Compared to the widely used facial recognition, this approach addresses the recent struggles of identifying faces behind masks and the public concerns about privacy erosion.

Behavioral biometrics has emerged as an important security factor for user authentication. Compared to static biometrics (e.g., faces, irises, and fingerprints), using human motion behaviors for authentication causes lower concern about privacy abuse, and behavior biometrics are shown hard to be replicated by humans. In-air 3D signature is one representative of behavioral biometrics. Specifically, a user’s hand movements can be tracked by visual or wireless sensors for contact-free signature authentication, where both the fingertip trajectory and the dynamic motion features are verified to provide enhanced security. However, with the advancement of 3D printing and robot technology, we find that 1) existing hand-tracking interfaces (e.g., Leap Motion and Google MediaPipe) are easily tricked by a fake hand, and 2) a robotic arm can reproduce a user’s in-air 3D signature with high similarity regarding both trajectory and motion behaviors. Thus, this work investigates the security of in-air signatures under robot-level replays and proposes to extend the signature verification from a single-point fingertip to multiple hand joints for enhanced security. We develop the hand skeleton-based 3D signature verification system, which can be deployed on any single camera devices (2D or 3D). The key insight is that current robots could hardly replicate the minute and unique inter-joint motions of a user. In particular, we track the hand skeleton using a single camera and reconstruct/draw the trajectories of its joints in a virtual 3D space, using the color gradients to represent time-lapse and using varying line widths to describe joint significance. Based on that, we extract the three-view skeleton signatures and inter-joint motion features and develop a convolutional neural network for verification. Extensive experiments show that our system not only achieves high authentication performance but also effectively mitigates robot-level replay attacks.

While Virtual Reality (VR) applications are becoming increasingly common, efficiently verifying a VR device user before granting personal access is still a challenge. Existing VR authentication methods require users to enter PINs or draw graphical passwords using controllers. Though the entry is in the virtual space, it can be observed by others in proximity and is subject to critical security issues. Furthermore, the in-air hand movements or handheld controller-based authentications require active user participation and are not time-efficient. This work proposes a low-effort VR device authentication system based on the unique skull-reverberated sounds, which can be acquired when the user wears the VR device. Specifically, when the user puts on the VR device or is wearing it to log into an online account, the proposed system actively emits an ultrasonic signal to initiate the authentication session. The signal returning to the VR device’s microphone has been reverberated by the user’s head, which is unique in size, skull shape and mass. We thus extract head biometric information from the received signal for unobtrusive VR device authentication.Though active acoustic sensing has been broadly used on mobile devices, no prior work has ever successfully applied such techniques to commodity VR devices. Because VR devices are designed to provide users with virtual reality immersion, the echo sounds used for active sensing are unwanted and severely suppressed. The raw audio before this process is also not accessible without kernel/hardware modifications. Thus, our work further solves the challenge of active acoustic sensing under echo cancellation to enable deploying our system on off-the-shelf VR devices. Additionally, we show that the echo cancellation mechanism is naturally good to prevent acoustic replay attacks. The proposed system is developed based on an autoencoder and a convolutional neural network for biometric data extraction and recognition. Experiments with a standalone and a mobile phone VR headset show that our system efficiently verifies a user and is also replay-resistant.

Biometrics have been widely applied as personally identifiable data for user authentication. However, existing biometric authentications are vulnerable to biometric spoofing. One reason is that they are easily observable and vulnerable to physical forgeries. Examples are the apparent surface patterns of human bodies, such as fingerprints and faces. A more significant issue is that existing authentication methods are entirely built upon biometric features, which almost never change and could be obtained or learned by an adversary such as human voices. To address this inherent security issue of biometric authentications, we propose a novel acoustically extracted hand-grip biometric, which is associated with every user’s hand geometry, body-fat ratio, and gripping strength; It is implicit and available whenever they grip a handheld device. Furthermore, we integrate a coding technique in the biometric acquisition process, which encodes static biometrics into dynamic biometric features to prevent data reuse. Additionally, this low-cost method can be deployed on any handheld device that has a speaker and a microphone. In particular, we develop a challenge-response biometric authentication system, which consists of a pair of biometric encoder and decoder. We encode the ultrasonic signal according to a challenge sequence and extract a distinct biometric code as the response for each session. We then decode the biometric code to verify the user by a convolutional neural network-based algorithm, which not only examines the coding correctness but also verifies the biometric features presented by each biometric digit. Furthermore, we investigate diverse acoustic attacks to our system, by respectively assuming an adversary could present the correct code, generate similar biometric features or successfully forge both. Extensive experiments on mobile devices show that our system achieves 97% accuracy to distinguish users and rejects 100% replay and synthesis attacks with 6-digit codes.

Motion-controlled robotic arms allow a user to interact with a remote real world without physically reaching it. By connecting cyberspace to the physical world, such interactive teleoperations are promising to improve remote education, virtual social interactions, and online participatory activities. In this work, we build up a motion-controlled robotic arm framework comprising a robotic arm end and a user end, which are connected via a network and responsible for manipulator control and motion capture, respectively. To protect the system access, we propose to verify who is controlling the robotic arm by examining the robotic arm’s behavior, which adds a second security layer in addition to the system login credentials. We show that a robotic arm’s motion inherits its human controller’s behavioral biometric in interactive control scenarios. By extracting the angle readings of the robotic arm’s all joints, the proposed user authentication approach reconstructs the robotic arm’s end-effector movement trajectory that follows the user’s hand. Furthermore, we derive the unique robotic motion features to capture the user’s behavioral biometric embedded in the robot motions and develop learning-based algorithms to verify the robotic arm user to be one of the enrolled users or a nonuser. Extensive experiments show that our system achieves 94% accuracy to distinguish users while preventing user identity spoofing attacks with 95% accuracy.

Handheld phone distraction is the leading cause of traffic accidents. However, few efforts have been devoted to detecting when the phone distraction happens, which is a critical input for taking immediate safety measures. This work proposes a phone-use monitoring system, which detects the start of the driver’s handheld phone use and eliminates the distraction at once. Specifically, the proposed system emits periodic ultrasonic pulses to sense if the phone is being held in hand or placed on support surfaces (e.g., seat and cup holder) by capturing the unique signal interference resulted from the contact object’s damping, reflection and refraction. We derive the short-time Fourier transform from the microphone data to describe such impacts and develop a CNN-based binary classifier to discriminate the phone use between the handheld and the handsfree status. Additionally, we design an adaptive window-based filter to correct the classification errors and identify each handheld phone distraction instance, including its start, end, and duration. Extensive experiments with fourteen people, three phones and two car models show that our system achieves 99% accuracy of recognizing handheld phone-use instances and 0.76-second median error to estimate the distraction’s start time.

This work proposes a media sound-based authentication method to protect smartphone notification privacy unobtrusively, which wisely hides or presents sensitive content by verifying who is holding the phone. We show that media sounds, such as the melodies of notification tones (e.g., iPhone message and Samsung whistle) can be directly used to sense and verify the user's gripping hand. Because sounds and vibrations co-exist, we capture two novel responses via the smartphone mic and accelerometer to describe how the individual's contacting palm interferes with the signals in two different domains. Based on the two responses, we develop a convolutional neural network-based algorithm to verify the user. Moreover, because the smartphone sensors are all embedded on the same motherboard, we develop a cross-domain method to validate such hard-to-forge physical relationships among the mic, speaker and accelerometer. They prevent external sounds from cheating the system. Additionally, we consider the notification vibration as a special type of media sound, which also results in two responses, and extend our method to work in the silent mode. Extensive experiments with ten notification tones and four phone models show that our system verifies users with 95% accuracy and prevents replay sounds with 100% accuracy.

Motion-controlled robots allow a user to interact with a remote real world without physically reaching it. By connecting cyberspace to the physical world, such interactive teleoperations are promising to improve remote education, virtual social interactions and online participatory activities. This work builds up a motion-controlled robotic arm framework and proposes to verify who is controlling the robotic arm by examining the robotic arm's behavior. We show that a robotic arm's motion inherits its human controller's behavioral biometric in interactive control scenarios. Furthermore, we derive the unique robotic motion features to capture the user's behavioral biometric embedded in the robot motions and develop learning-based algorithms to verify the robotic arm user. Extensive experiments show that our system achieves high accuracy to distinguish users while using the robot's behaviors.

Phone usage while driving is unanimously considered a really dangerous habit due to a strong correlation with road accidents. This paper proposes a phone-use monitoring system that detects the driver's handheld phone use and eliminates the distraction at once. Specifically, the proposed system emits periodic ultrasonic pulses to sense if the phone is being held in hand or placed on support surfaces (e.g., seat and cup holder) by capturing the unique signal interference resulted from the contact object's damping, reflection and refraction. We derive the short-time Fourier transform from the microphone data to describe such impacts and develop a CNN-based binary classifier to discriminate the phone use between the handheld and the handsfree status. Additionally, we design a classification error correction filter to correct the classification errors during the monitoring. The experiments with six people, one phone and one car model show that our system achieves 99% accuracy in recognizing handheld phone-use activities.

As the most common personal devices, smartphones contain the user's private information. While people use mobile devices anytime and anywhere, the sensitive contents might be leaked from the screens. The smartphone notifications cause such privacy leakages even on a lock screen. With the aim to alert the user of an event (e.g., text messages, phone calls and calendar reminders), these onscreen notifications usually contain the sender's name and even a clip of the contents for preview. Such information, if not displayed appropriately, may cause the leakages of the user's social relations, personal hobbies and private message contents. This work focuses on wisely displaying the notifications to avoid leaking the user's privacy. We develop an unobtrusive user authentication system to confirm the user identity via their gripping-hands before displaying notifications. In particular, we carefully design an inaudible acoustic signal and emit it from the smartphone speaker to sense the gripping hand, when there is a need to push notifications. The signal propagating to the smartphone's microphones carries the user's biometric information related to the gripping hand (e.g., palm size and gripping strength). We further derive the Mel Frequency Cepstral Coefficient time series and develop a machine learning-based algorithm to identify the user. The experimental results show that our system can identify 8 users with 92% accuracy.