With the rising of AR/VR technology and miniaturization of mobile devices, gesture recognition is becoming increasingly popular in the research area of human-computer interaction. Some pioneer ultrasound-based gesture recognition systems have been proposed. However, they mostly rely on low-resolution Doppler Effect, with the focus on whole hand motion and fail to deal with minor finger motions. This paper is to present UltraGesture, an ultrasonic finger motion perception and recognition system based on Channel Impulse Response (CIR). CIR measurements can provide with 7 mm resolution, which is sufficient for minor finger motion recognition. UltraGesture encapsulates CIR measurements into image, and builds a Convolutional Neural Network model to classify these images into different categories corresponding to distinct gestures. Furthermore, we use a sliding-window based method to improve accuracy and reduce response latency. UltraGesture can run on the already existed commercial speakers and microphones on most mobile devices without any hardware modification. Our results demonstrate that UltraGesture can achieve an average accuracy ofgreater than 99 percent for 12 gestures including finger click and rotation.

Cellular network operators deploy base stations with a high density to ensure radio signal coverage for 4G/5G networks. While users enjoy the high-speed connection provided by cellular networks, an adversary could exploit the dense cellular deployment to detect nearby human movements and even recognize keystroke movements of a victim by passively listening to the CRS broadcast from base stations. To demonstrate this, we develop SpiderMon, the first attempt to perform passive continuous keystroke monitoring using the signal transmitted by commercial cellular base stations. Our experimental results show that SpiderMon can detect keystroke movements at a distance of 15 meters and can recover a 6-digits PIN input with a success rate of more than 51% within ten trials when the victim is behind the wall.

Voice has been used as biometrics for human authentication because different people have different voice characteristics due to different vocal tract shapes and intonations. However, traditional voice based human authentication is subject to four types of attacks: impersonation, voice conversion, synthesis and voice replay. In this paper, we propose SpeakPrint, an ultrasound based human speech authentication scheme for smartphones which is resistant for these attacks. Compared with traditional speech authentication system which focuses on what a user speaks, SpeakPrint captures how a user speaks by recording mouth and vocal movement through ultrasound signal at the same time. Our key insight is that for the valid user, features extracted from voice signal should be consistent with his mouth and vocal movement recorded from ultrasound signal, while an imitator or an audio player can't produce the same signals in ultrasound domain. SpeakPrint extracts MFCC feature in normal voice frequency and MMSI features from ultrasound signal. An SVM classifier is trained to detect these attacks by comparing above feature differences. We implemented SpeakPrint on Samsung S5 and conducted experiments on 40 users. Experimental results show that SpeakPrint can detect replay attacks with 100% accuracy and replay attack with lip synching for 99.12% for passphrases longer than five words. This technology can be used in multi-factor authentication systems, where multiple authentication mechanisms are used to achieve defense in depth.

With the rising of AR/VR technology and miniaturization of mobile devices, gesture is becoming an increasingly popular means of interacting with smart devices. Some pioneer ultrasound based human gesture recognition systems have been proposed. They mostly rely on low resolution Doppler Effect, and hence focus on whole hand motion and cannot deal with minor finger motions. In this paper, we present UltraGesture, a Channel Impulse Response (CIR) based ultrasonic finger motion perception and recognition system. CIR measurements can provide with 7 mm resolution, rendering it sufficient for minor finger motion recognition. UltraGesture encapsulates CIR measurements into an image, and builds a Convolutional Neural Network model to classify these images into different categories, which corresponding to distinct gestures. Our system runs on commercial speakers and microphones that already exist on most mobile devices without hardware modification. Our results show that UltraGesture achieves an average accuracy of greater than 97% for 12 gestures including finger click and rotation.

Since human bodies are good reflectors of wireless signals, human activities can be recognized by monitoring changes in WiFi signals. However, existing WiFi-based human activity recognition systems do not build models that can quantify the correlation between WiFi signal dynamics and human activities. In this paper, we propose a Channel State Information (CSI)-based human Activity Recognition and Monitoring system (CARM). CARM is based on two theoretical models. First, we propose a CSI-speed model that quantifies the relation between CSI dynamics and human movement speeds. Second, we propose a CSI-activity model that quantifies the relation between human movement speeds and human activities. Based on these two models, we implemented the CARM on commercial WiFi devices. Our experimental results show that the CARM achieves recognition accuracy of 96% and is robust to environmental changes.

Some pioneer WiFi signal based human activity recognition systems have been proposed. Their key limitation lies in the lack of a model that can quantitatively correlate CSI dynamics and human activities. In this paper, we propose CARM, a CSI based human Activity Recognition and Monitoring system. CARM has two theoretical underpinnings: a CSI-speed model, which quantifies the correlation between CSI value dynamics and human movement speeds, and a CSI-activity model, which quantifies the correlation between the movement speeds of different human body parts and a specific human activity. By these two models, we quantitatively build the correlation between CSI value dynamics and a specific human activity. CARM uses this correlation as the profiling mechanism and recognizes a given activity by matching it to the best-fit profile. We implemented CARM using commercial WiFi devices and evaluated it in several different environments. Our results show that CARM achieves an average accuracy of greater than 96%.