As acoustic communication systems become increasingly common in our daily life, eavesdropping brings severe security and privacy risks. Current methods of acoustic eavesdropping either provide low resolution due to the use of sub-6 GHz frequencies, work only for limited words based on classification approaches, or cannot work through-wall because of the use of optical sensors. In this article, we presentmilliEar, a mmWave acoustic eavesdropping system that leverages the high-resolution of mmWave FMCW ranging and generative machine learning models to not only extract vibrations but to reconstruct the audio.milliEarcombines speaker vibration estimation with conditional generative adversarial networks to eavesdrop and recover high-quality audios (i.e., with no vocabulary constraints). We implement and evaluatemilliEarusing off-the-shelf mmWave radars deployed in different scenarios and settings. Evaluation results clearly show thatmilliEarcan accurately reconstruct the audio even at different distances, angles, and through the wall with different insulator materials. In addition, our subjective and objective evaluations demonstrate that the reconstructed audio has a strong similarity with the original audio.

As acoustic communication systems become more common in homes and offices, eavesdropping brings significant security and privacy risks. Current approaches of acoustic eavesdropping either provide low resolution due to the use of sub-6 GHz frequencies, work only for limited words using classification, or cannot work through-wall due to the use of optical sensors. In this paper, we present MILLIEAR, a mmWave acoustic eavesdropping system that leverages the high-resolution of mmWave FMCW ranging and generative machine learning models to not only extract vibrations but to reconstruct the audio. MILLIEAR combines speaker vibration estimation with conditional generative adversarial networks to eavesdrop with unconstrained vocabulary. We implement and evaluate MIL-LIEAR using off-the-shelf mmWave radar deployed in different scenarios and settings. We find that it can accurately reconstruct the audio even at different distances, angles and through the wall with different insulator materials. Our subjective and objective evaluations show that the reconstructed audio has a strong similarity with the original audio.

With the increasing popularity of voice-based applications, acoustic eavesdropping has become a serious threat to users’ privacy. While on smartphones the access to microphones needs an explicit user permission, acoustic eavesdropping attacks can rely on motion sensors (such as accelerometer and gyroscope), which access is unrestricted. However, previous instances of such attacks can only recognize a limited set of pre-trained words or phrases. In this paper, we present AccEar, an accelerometer-based acoustic eavesdropping attack that can reconstruct any audio played on the smartphone’s loudspeaker with unconstrained vocabulary. We show that an attacker can employ a conditional Generative Adversarial Network (cGAN) to reconstruct high-fidelity audio from low-frequency accelerometer signals. The presented cGAN model learns to recreate high-frequency components of the user’s voice from low-frequency accelerometer signals through spectrogram enhancement. We assess the feasibility and effectiveness of AccEar attack in a thorough set of experiments using audio from 16 public personalities. As shown by the results in both objective and subjective evaluations, AccEar successfully reconstructs user speeches from accelerometer signals in different scenarios including varying sampling rate, audio volume, device model, etc.

Gestures in American Sign Language (ASL) are characterized by fast, highly articulate motion of upper body, including arm movements with complex hand shapes and facial expressions. In this work, we propose a new method for word-level sign recognition from American Sign Language (ASL) using video. Our method uses both motion and hand shape cues while being robust to variations of execution. We exploit the knowledge of the body pose, estimated from an off-the-shelf pose estimator. Using the pose as a guide, we pool spatio-temporal feature maps from different layers of a 3D convolutional neural network. We train separate classifiers using pose guided pooled features from different resolutions and fuse their prediction scores during test time. This leads to a significant improvement in performance on the WLASL benchmark dataset [25]. The proposed approach achieves 10%, 12%, 9.5% and 6.5% performance gain on WLASL100, WLASL300, WLASL1000, WLASL2000 subsets respectively. To demonstrate the robustness of the pose guided pooling and proposed fusion mechanism, we also evaluate our method by fine tuning the model on another dataset. This yields 10% performance improvement for the proposed method using only 0.4% training data during fine tuning stage.

This work presents an approach for American Sign Language (ASL) gesture recognition from videos. Gestures are comprised of various upper body motions involving hand shapes, motion of both hands with facial expression and head movements. Previous approaches tackled this problem by directly learning 3D convolutional spatio-temporal models from video in a simplified settings with uniform backgrounds. To handle more complex variation in appearance and backgrounds we propose to exploit recent advances in estimation of 2D body pose using Deep Convolutional Neural Networks trained on large corpus of human pose annotations. We use the trajectories of 2D skeletal data estimated from video to train a baseline recursive neural network gesture recognition model. The basic model is further extended using embeddings of hand images obtained from another hand shape recognition model [15] with dynamics modeled by another recursive neural network. The final model learns how to fuse two Long Short Term Model (LSTM) recursive neural network models for skeletal and hand image data. We train and evaluate this model on the GMU-ASL51 dataset of 12 users and 51 ASL gestures [8] demonstrating its superior performance compared to several baseline models.

American Sign Language recognition is a difficult gesture recognition problem, characterized by fast, highly articulate gestures. These are comprised of arm movements with different hand shapes, facial expression and head movements. Among these components, hand shape is the vital, often the most discriminative part of a gesture. In this work, we present an approach for effective learning of hand shape embeddings, which are discriminative for ASL gestures. For hand shape recognition our method uses a mix of manually labelled hand shapes and high confidence predictions to train deep convolutional neural network (CNN). The sequential gesture component is captured by recursive neural network (RNN) trained on the embeddings learned in the first stage. We will demonstrate that higher quality hand shape models can significantly improve the accuracy of final video gesture classification in challenging conditions with variety of speakers, different illumination and significant motion blurr. We compare our model to alternative approaches exploiting different modalities and representations of the data and show improved video gesture recognition accuracy on GMU-ASL51 benchmark dataset.

Over half a million people in the United States use American Sign Language (ASL) as their primary mode of communication. Automatic ASL recognition would enable Deaf and Hard of Hearing (DHH) users to interact with others who are not familiar with ASL as well as voice-controlled digital assistants (e.g., Alexa, Siri, etc.). While ASL recognition has been extensively studied, there is a little attention given to recognition of ASL non-manual body markers. The non-manual markers are typically expressed through head, torso and shoulder movements, and add essential meaning and context to the signed sentences. In this work, we present ExASL, a sentence-level ASL recognition system using millimeter-wave radars. ExASL can recognize manual markers (hand gestures) and non-manual markers (head and torso movements). It utilizes multi-distance clustering to recognize body parts and cluster mmWave point clouds. We then present a multi-view deep learning algorithm that can learn from clustered body part representation for an expressive sentence-level recognition. Our evaluation shows that ExASL can recognize ASL sentences with a word error rate of 0.79%, sentence error rate of 1.25%, and non-manual markers with an accuracy of 83.5%.

Voice-controlled personal and home assistants (such as the Amazon Echo and Apple Siri) are becoming increasingly popular for a variety of applications. However, the benefits of these technologies are not readily accessible to Deaf or Hard-of-Hearing (DHH) users. The objective of this study is to develop and evaluate a sign recognition system using multiple modalities that can be used by DHH signers to interact with voice-controlled devices. With the advancement of depth sensors, skeletal data is used for applications like video analysis and activity recognition. Despite having similarity with the well-studied human activity recognition, the use of 3D skeleton data in sign language recognition is rare. This is because unlike activity recognition, sign language is mostly dependent on hand shape pattern. In this work, we investigate the feasibility of using skeletal and RGB video data for sign language recognition using a combination of different deep learning architectures. We validate our results on a large-scale American Sign Language (ASL) dataset of 12 users and 13107 samples across 51 signs. It is named as GMU-ASL51. We collected the dataset over 6 months and it will be publicly released in the hope of spurring further machine learning research towards providing improved accessibility for digital assistants.

Dense deployment of access points in 60 GHz WLANs can provide always-on gigabit connectivity and robustness against blockages to mobile clients. However, this dense deployment can lead to harmful interference between the links, affecting link data rates. In this paper, we attempt to better understand the interference characteristics and effectiveness of interference mitigation techniques using 802.11ad COTS devices and 60 GHz software radio based measurements. We first find that current 802.11ad COTS devices do not consider interference in sector selection, resulting in high interference and low spatial reuse. We consider three techniques of interference mitigation - channelization, sector selection and receive beamforming. First, our results show that channelization is effective but 60 GHz channels have non-negligible adjacent and non-adjacent channel interference. Second, we show that it is possible to perform interference-aware sector selection to reduce interference but its gains can be limited in indoor environment with reflections, and such sector selection should consider fairness in medium access and avoid asymmetric interference. Third, we characterize the efficacy of receive beamforming in combating interference and quantify the related overhead involved in the search for receive sector, especially in presence of blockages. We elaborate on the insights gained through the characterization and point out important outstanding problems through the study.