RageSense introduces a novel system for detecting and regulating player frustration during mobile gaming. Instead of relying on coarse emotion labels, RageSense estimates users’ valence and arousal levels in real time using near-ultrasonic acoustic sensing. By analyzing facial muscle movements via built-in smartphone speakers and microphones, our approach enables emotion sensing without requiring cameras or wearables, constituting a more unobtrusive, environment-resilient, and privacy-friendly approach than traditional emotion recognition. To transform detection into action, we integrate a large language model (LLM) that generates empathetic, context-aware interventions based on gameplay screenshots, behavioral signals, and emotional trajectories. These interventions are delivered in real time, tailored to the user’s emotional state, and designed to mitigate rage while enhancing player well-being. In a 53-participant field study, our system improved emotional state immediately after triggers and was preferred over random or template-based messages. To our knowledge, this is the first demonstration of near-ultrasonic, on-phone valence-arousal regression during mobile gameplay that directly drives real-time, context-aware interventions.

Silent speech interface (SSI) enables users to interact with their devices without making audible sounds, thus preventing potential eavesdropping or disruptions to others. Recent advancements in acoustic sensing technology have made SSI on mobile devices highly promising, requiring no hardware modifications and operating in a non-contact manner. However, one major challenge faced by existing acoustic sensing-based SSI is its limited sensing range, typically less than 7cm. Users often need to speak in close proximity to the speakers/microphones, severely constraining its applicability on mobile devices. In this paper, we introduce EchoLip, which can significantly increase the sensing range and enhance long-term usability in real-world settings. EchoLip utilizes the smartphone’s two built-in speakers and microphones to transmit and receive mutually orthogonal wave signals to capture multi-view information. Then, a specially designed signal processing pipeline and neural network are used to extract fine-grained features that adapt to different angles and distances. We also design a lip movement monitoring algorithm to handle various interference. We evaluate EchoLip on 20 individuals using a set of 500 sentences. EchoLip achieves an average Word Error Rate of 13.9% and 19.7% at 15cm and 40cm. Evaluations in various scenarios further validate the robustness of EchoLip.

Accurately recognizing facial expressions and emotions at any time and in any place can significantly improve people's quality of life and mental well-being. However, existing methods lack the convenient capability for long-term monitoring in the wild environment. In this paper, we introduce EchoExpress, an in-the-wild emotion-related facial expression recognition system that works in an unobtrusive, low-power, and privacy-friendly way. EchoExpress uses two speakers and two microphones mounted on a glass-frame for transmitting and receiving mutually orthogonal wave signals. Concurrently, a unique attention mechanism dynamically extracts crucial features, enabling the capture of nuanced facial expressions and emotions. Furthermore, we introduce an open-set filtering mechanism with a specially designed loss function, which effectively filters out irrelevant actions, thereby reducing the risk of misidentification. Finally, a semi-supervised training method is employed to address the significant variability in wild expressions across different individuals. In extensive testing, EchoExpress achieves an accuracy of 84% in a laboratory environment and over 75% in real-world conditions. We believe that EchoExpress can serve as an unobtrusive and reliable way to monitor facial expressions.

Wireless-based gesture recognition provides an effective input method for exergames. However, previous works in wireless-based gesture recognition systems mainly recognize one primary user's gestures. In the multi-player scenario, the mutual interference between users makes it difficult to predict multiple players' gestures individually. To address this challenge, we propose a flexible FMCW-radar-based system, RFDual, which enables real-time cross-domain gesture sequence recognition for two players. To eliminate the mutual interference between users, we extract a new feature type, biased range-velocity spectrum (BRVS), which only depends on a target user. We then propose customized preprocessing methods (cropping and stationary component removal) to produce environment-independent and position-independent inputs. To enhance RFDual's resistance to unseen users and articulating speeds, we design effective data augmentation methods, sequence concatenating, and randomizing. RFDual is evaluated with a dataset containing only unseen gesture sequences and achieves a gesture error rate of 1.41%. Extensive experimental results show the impressive robustness of RFDual for data in new domains, including new users, articulating speeds, positions, and environments. These results demonstrate the great potential of RFDual in practical applications like two-player exergames and gesture/activity recognition for drivers and passengers in the cab.

With deep learning, millimeter-wave radar-based gesture recognition applications have achieved satisfactory results. However, most existing approaches highly rely on highquality labeled data, and they suffer from severe over-fitting when labeled data are scarce. To end this, we present RadarAE, a novel representation learning framework for radar sensing applications. RadarAE learns sophisticated representations from massive low-cost unlabeled radar data, which enables accurate gesture recognition with few labeled data. To achieve this goal, we first meticulously observe the characteristics of raw radar data and extract an effective feature, Spatio-Temporal Motion Map (STMM). Then we borrow the key principle of Masked Autoencoders (MAE), a self-supervised learning technique for images, and propose an MAE-like model to learn useful representations from STMM. To adapt RadarAE to radar sensing applications, we present a series of customization techniques, including data augmentation, optimized model structure, and adaptive pretraining method. With the learned high-level representations, gesture recognition models can achieve superior performance in few-shot scenarios. Experiment results show that our model can achieve 79.1%, 92.1%, 97.8%, and 99.5% recognition accuracy in the 1, 2, 4, and 8-shot scenarios, respectively, where x-shot refers to the number of labeled samples for each gesture type. The source codes and dataset are made publicly available11https://githuh.com/Ela-Boska/RadarAE.

In modern society, wireless signals are ubiquitous in various indoor environments, such as living houses, offices and shop malls, facilitating human living in various aspects. Action recognition is a technique in roaring demand in the field of human-computer interaction. Whereas previous research studies propose various methods to action recognition using wireless signals, action recognition in locations with limited data is still very challengeable. To realize decent action recognition with the help of wireless signals, we propose WiLAR, a location-adapted action recognition system based on WiFi, which enables action detection, segmentation and recognition with commodity WiFi devices in locations with different amounts of data. WiLAR extracts informative features from fine-grained WiFi channel state information (CSI), and then feeds features into elaborately designed deep learning models to realize action recognition in different locations. In our dedicated experiments, WiLAR achieves average 97% accuracy workout recognition in locations with plenty of data, and also outperforms other recognition models in locations with limited training data.

Handwritten signature verification techniques, which can facilitate user authentication and enable secure information exchange, are still important in property safety. However, on-line automatic handwritten signature verification usually requires dynamic handwritten patterns captured by a special device, such as a sensor-instrumented pen, a tablet or a smartwatch on the dominant hand. This paper presents SonarSign, an on-line handwritten signature verification system based on inaudible acoustic signals. The key insight is to use acoustic signals to capture the dynamic handwritten signature patterns for verification. Particularly, SonarSign exploits the built-in speakers and microphones of smartphones to transmit a specially designed training sequence and record the corresponding echo for channel impulse response (CIR) estimation, respectively. Based on the sensitivity of CIR to the tiny surrounding environment changes including handwritten signature actions, SonarSign designs an attentional multi-modal Siamese network for end-to-end signatures verification. First, multi-modal CIR streams are fused to extract representative signature pattern features from spatio-temporal dimensions. Then an attentional Siamese network is elaborated to verify whether the given two signatures are from the same signatory. Extensive experiments in real-world scenarios show that SonarSign can achieve accurate and robust signatures verification with an AUC (Area Under ROC (Receiver Operating Characteristic) Curve) of 98.02% and an EER (Equal Error Rate) of 5.79% for unseen users.

Gesture recognition plays a fundamental role in emerging Human-Computer Interaction (HCI) paradigms. Recent advances in wireless sensing show promise for device-free and pervasive gesture recognition. Among them, RFID has gained much attention given its low-cost, light-weight and pervasiveness, but pioneer studies on RFID sensing still suffer two major problems when it comes to gesture recognition. The first is they are only evaluated on simple whole-body activities, rather than complex and fine-grained hand gestures. The second is they can not effectively work without retraining in new domains, i.e. new users or environments. To tackle these problems, in this paper, we propose RFree-GR, a domain-independent RFID system for complex and fine-grained gesture recognition. First of all, we exploit signals from the multi-tag array to profile the sophisticated spatio-temporal changes of hand gestures. Then, we elaborate a Multimodal Convolutional Neural Network (MCNN) to aggregate information across signals and abstract complex spatio-temporal patterns. Furthermore, we introduce an adversarial model to our deep learning architecture to remove domain-specific information while retaining information relevant to gesture recognition. We extensively evaluate RFree-GR on 16 commonly used American Sign Language (ASL) words. The average accuracy for new users and environments (new setup and new position) are $89.03%$, $90.21%$ and $88.38%$, respectively, significantly outperforming existing RFID based solutions, which demonstrates the superior effectiveness and generalizability of RFree-GR.

Smart retail stores open new possibilities for enabling a variety of physical analytics, such as users' shopping trajectories and preferences for certain items. This paper aims to excavate three kinds of relations in physical stores, i.e. user-item, user-user and item-item, which provide abundant information for enhancing users' shopping experiences and boosting retailers' sales. We present ShopEye, a hybrid RFID and smartwatch system to delve into these relations in an implicit and non-intrusive manner. The intuition is that inertial sensors embedded in smartwatches and RFID tags attached to items can capture the user behaviors and the item motions, respectively. ShopEye first pairs users with corresponding items according to correlations between inertial signals and RFID signals, and then incorporates these pairs with the motion behaviors of users to further profile user-user and item-item relations. We have tested the system extensively in our lab environment which mimics the real retail store. Experimental results demonstrate the effectiveness and robustness of ShopEye in excavating these relations.

Automatic sign language recognition is an important milestone in facilitating the communication between the deaf community and hearing people. Existing approaches are either intrusive or susceptible to ambient environments and user diversity. Moreover, most of them perform only isolated word recognition, not sentence-level sequence translation. In this paper, we present MyoSign, a deep learning based system that enables end-to-end American Sign Language (ASL) recognition at both word and sentence levels. We leverage a lightweight wearable device which can provide inertial and electromyography signals to non-intrusively capture signs. First, we propose a multimodal Convolutional Neural Network (CNN) to abstract representations from inputs of different sensory modalities. Then, a bidirectional Long Short Term Memory (LSTM) is exploited to model temporal dependences. On the top of the networks, we employ Connectionist Temporal Classification (CTC) to get around temporal segments and achieve end-to-end continuous sign language recognition. We evaluate MyoSign on 70 commonly used ASL words and 100 ASL sentences from 15 volunteers. Our system achieves an average accuracy of 93.7% at word-level and 93.1% at sentence-level in user-independent settings. In addition, MyoSign can recognize sentences unseen in the training set with 92.4% accuracy. The encouraging results indicate that MyoSign can be a meaningful buildup in the advancement of sign language recognition.

Regular exercise offers numerous health benefits and contributes to a healthy lifestyle. Doing exercise at home is an attractive choice for many people due to its convenience and low cost. Motivated by this, we propose FitAssist in this paper, a household virtual fitness assistant capable of performing fine-grained exercise recognition and exercise quality assessment based on commercial WiFi devices. Unlike wearable devices based systems, this system is more comfortable and device-free. In addition, compared to previous Wi-Fi based exercise monitoring systems, whose performance attenuates seriously when users stand out of the First Fresnel Zone (FFZ), FitAssist does not require users to stand on or near the line of sight (LoS) path. To achieve this, FitAssist extracts features from the fine-grained WiFi channel state information (CSI) and enables both exercise recognition and user identification via deep learning techniques. Moreover, FitAssist can provide personalized workout assessment to help users obtain effective workout and prevent injury. Extensive experimental results in real settings show that FitAssist achieves average accuracies of 97% and 98% for exercise recognition and user identification respectively, as well as giving accurate and useful feedback in various scenarios, which proves its effectiveness and robustness.

Monitoring aperiodic activity sequence contributes a lot to home exercise guidance and sports experience but existing approaches are designed for quasi-period activity or isolated activity monitoring. There is a compelling need for contactless real-time auxiliary exercise system, especially for aperiodic activity sequence. In this paper, we present Wiga, a WiFi-based real-time contactless activity sequence recognition system, which can recognize activity sequences even for users who have not participated in the training phase. Wiga takes the fine-grained Channel State Information (CSI) as input and elaborates a deep learning network to map the motion-induced signal variations with the activity sequence. First, Wiga removes noise and redundancy of the raw CSI measurements. Then, after abstracting deep features with a Convolutional Neural Network (CNN), Wiga exploits a Long Short Term Memory (LSTM) network to model temporal dependencies of the sequence. In addition, Wiga employs the beam search method to get around error-prone temporal segments and obtains real-time activity sequence recognition. We evaluate Wiga with 17 yoga activities from 7 volunteers, and extensive experimental results show that Wiga achieves an average accuracy of 97.7% and 85.6% for trained and untrained users respectively with a recognition delay no more than 0.5s.

Smart wearable devices have become pervasive as they are portable and intelligent. The popular method to interact with it is touch-screen, which is error-prone and cumbersome due to its limited size. There are a few innovative works designing a virtual dial plate on the hand back, which need special-purpose sensors or microphones which may suffer from privacy leak. We propose MType, a system only employs sensors already built in the commercial-off-the-shelf (COTS) device with a magnetic ring to expand the interaction space between users and wearable devices. The core idea is to leverage the gravity sensor, linear accelerometer and magnetometer embedded in the standard smartwatch to detect gestures, capture input events and locate keystrokes on the opisthenar and palm. Besides, MType designs a runtime adaptation mechanism to handle the cold start problem and adapt to the variations over the time of usage. We implement MType on the COTS smartwatch and our extensive experiments in different scenarios show that the average accuracy of keystroke localization can reach 93% with a small size initial training set (3 samples for each key) at a low sampling rate (51Hz). Furthermore, when turning on the runtime adaptation mechanism and enlarging the training set, the accuracy can achieve 98%.

In recent years, location-based services have been widely applied not only in daily life but also in automation industries. As one of main location sensing technologies, RFID based localization has attracted increasing attention. Existing synthetic aperture RFID localization systems use the inverse correlation filter to reconstruct holograms and achieve satisfactory accuracy. However, these methods require accurate aperture positions for theoretical signal construction, while the ubiquitous aperture uncertainty in practice causes non-negligible performance degradation. In this paper, we present PEC, an accurate synthetic aperture RFID localization system with aperture position error compensation, which has a major advantage over the classic systems for no need to know the exact trajectory of the synthetic aperture. We first build a mathematical model for localization and merge all coherent received signals to estimate the tag position. Then we propose an iterative algorithm which can alternately estimate both the tag position and the aperture position error. We have implemented and evaluated PEC using commercial-off-the-shelf (COTS) RFID devices. Extensive experimental results show that it achieves the cm-level accuracy with aperture position error in noisy environments, which proves its effectiveness and robustness.

In recent years, radio frequency identification (RFID)-based approaches have been demonstrated to be a promising indoor localization techniques for many valuable applications, such as tracking tagged objects on the manufacturing lines, locating items in smart warehouses, and so on. In the near future, many applications will gain great benefits from knowing the positions of RFID-tagged objects. However, existing localization approaches often suffer from severe accuracy degradation in real-world environments due to the prevalent environmental interferences, such as the multipath effects. To this end, we designed an RFID-based localization system FaHo, which leverages a deep learning enhanced holographic technique for locating RFID tags accurately even in complex indoor environments. By carefully analyzing the features of the traditional holographic method, we created a new hologram-based algorithm called joint hologram, which yields a robust likelihood for each assumed position to be the true tag position. FaHo then adopts a deep convolutional neural network for analyzing the whole hologram, and subsequently estimate the true location of the RFID tag rather than simply seek for the largest-likelihood location. Furthermore, we implemented FaHo and evaluated its performance in several multipath-rich scenarios. The experimental results show that FaHo can achieve centimeter-level accuracy in both the lateral and radial directions using only one moving antenna. More importantly, our work also demonstrates that hologram-based localization is a highly effective technique for RFID indoor localization tasks.

At present, wireless sensing based gesture recognition is becoming a rising star due to its convenience and non-invasiveness without privacy issues, while the strict requirement of the deployment and surrounding environment is still an unavoidable issue which limits its development and generalization. Although there are some works involving the environmental variance, the changes of relative positions between devices and users are ignored. As one of the most popular wireless sensing methods, RFID is widely used in activity recognition with its stable low-level physical characters such as phase and RSS. Besides, the signals reflected from RFID tags intuitively delineate its movements. On the other hand, many interactive gesture-driven applications, such as gesture input for video games, have a paramount and unavoidable issue about the latency between completion of a gesture and its recognition. Inspired by deep learning, this paper presents a real-time ongoing gesture recognition system EUIGR, which efficiently integrates phase and RSS data streams, and extracts both environment and user invariant features. The proposed system seamlessly integrates CNNs (Convolutional Neural Networks) and LSTM (Long Short-Term Memory) to fuse RFID low-level physical characters and extract space-temporal information. Furthermore, with adversarial learning, EUIGR suppresses environment-related factors and the user-specific features, and obtains strong robustness to individual diversity and decreases the environmental dependence. We also implement the system with COTS RFID devices, and extensive experimental results show the effectiveness and accuracy of EUIGR.

Online bookstores are capable of capturing readers preferences by analyzing click logs and transaction records, while physical bookstores or libraries still lack effective methods to gather reader behavioral data. Fortunately, the widespread use of mobile wearable devices and RFID technology opens up new possibilities for uncovering in-store experience. In this paper, we propose ReaderTrack, a system that integrates smartwatch and RFID to excavate interactions between readers and books. We first leverage inertial sensors of smartwatch and backscatter signals of RFID tags to infer reader behaviors and book motions, respectively. Then we associate readers with their corresponding books according to previously inferred behaviors and motions. We implement ReaderTrack with COTS devices and evaluate it extensively in our lab environment which mimics a typical reading room. Experimental results show the effectiveness and robustness of ReaderTrack in reader-book interaction reasoning.

In the future, libraries and warehouses will gain benefits from the spatial location of books and merchandises attached with RFID tags. Existing localization algorithms, however, usually focus on improving positioning accuracy or the ordering one for RFID tags on the same layer. Nevertheless, books or merchandises are placed on the multilayer in reality and the layer of RFID tagged object is also an important position indication. To this end, we design PRMS, an RFID based localization system which utilizes both phase and RSSI values of the backscattered signal provided by a single antenna to estimate the spatial position for RFID tags. Our basic idea is to gain initial estimated locations of RFID tags through a basic model which extracts the phase differences between received signals to locate tags. Then an advanced model is proposed to improve the positioning accuracy combined with RF hologram based on basic model. We further change traditional deployment of a single antenna to distinguish the features of RFID tags on multilayer and adopt a machine learning algorithm to get the layer information of tagged objects. The experiment results show that the average accuracy of layer detection and sorting at low tag spacing ($2\sim8$cm) are about 93% and 84% respectively.

2018 IEEE International Conference on Pervasive Computing and Communications (PerCom), Athens, Greece, March 19-23, 2018

Monitoring respiration and heartbeat contributes to disease prediction, sub-health diagnosis, exercise and sleep quality analysis, fatigue warning, and even emotion estimation. There is a compelling need for contactless, easy-to-deploy and long-term respiration and heartbeat monitoring. In this paper, we present CRH, an RFID-based contactless respiration and heartbeat monitoring system. The key insight is that the RFID signal fluctuation induced by the chest motion is synchronous with respiration and heartbeat. Therefore, CRH collects the temporal phase information from the tag array near or on body to extract respiration and heartbeat signals using a sequence of signal processing techniques. We propose a signal separation method based on multi-tag empirical mode decomposition (EMD) to obtain respiration rate and heart rate after preprocessing. Furthermore, CRH can also detect intense motions and abnormal respiration. We implement and evaluate CRH using Commercial Off-The-Shelf (COTS) RFID devices. Extensive experimental results in different scenarios show that CRH can achieve high accuracy for monitoring multi-user respiration and heart rates, validating its wide applicability and high reliability for contactless fine-grained respiration and heartbeat monitoring.

Gesture recognition is an innovative technology which is fundamentally reshaping the way people live, entertain and work. However, most gesture recognition systems focus on the recognition of simple gestures and ignore the full potential of sequential gestures involving a series of temporally-related simple actions in order. This paper presents SGRS, a battery-free, scalable and non-specific sequential gesture recognition system based on COTS RFID. The key insight is that finegrained phase information extracted from RF signals is capable of perceiving various gestures. In SGRS, we meticulously devise gesture recognition mechanism by incorporating the k-means based vector quantizer and string matching algorithm to enable precise and real-time sequential gesture identification. Moreover, an improved edit distance algorithm is proposed for suppressing individual diversity. We implement SGRS and comprehensively evaluate the performance by recognizing traffic command gestures of Chinese traffic police. Experimental result shows that SGRS achieves an average recognition accuracy of 96.2% with eight sequential gestures and is highly robust to both individual diversity and multipath effect.

Innovative Human Machine Interface technologies are fundamentally reshaping the way people live, entertain and work. Passive RFID tags, benefiting from its wireless, inexpensive and battery-free sensing ability, are gradually being applied in new-style interaction interfaces, ranging from virtual touch screen to 3D mouse. This paper presents TagController, a universal wireless and battery-free remote controller with two types of interactive actions. The key insight is that the fine-grained phase information extracted from RF signals is capable of perceiving various actions. TagController can recognize 10 actions without any training or prestored profiles by executing a sequence of functional components, i.e. preprocessor, action detector and action recognizer. We have implemented TagController with COTS RFID devices and conducted substantial experiments in different scenarios. The results demonstrate that TagController can achieve an average recognition accuracy of 95.8% and 94.3% in the scenarios of one and two remote controllers, respectively, which promises its feasibility and robustness.

Workflow recognition is a key technique in the field of activity recognition with benefits of monitoring the step being performed in the workflow, detecting the missing step, and providing assistance to the performer of the workflow, among others. In this paper, we present an unobtrusive workflow recognition system called RFlow-ID, which is the first device-free, battery-free and privacy-preserving workflow recognition system based on RFID technique. RFlow-ID perceives the use and movement of associated objects in the workflow using fine-grained phase information extracted from low-level RF signal, and infers the most likely sequence of workflow activities via a VQ-HMM model. We implement RFlow-ID on COTS RFID devices and evaluate it through a common biomedical experiment. The results validate the high recognition accuracy and robustness of our system.

Internet of Things (IoT) is rather prevalent in many manufacturing and smart city applications, while localization is a premise for many other processes, varying from ordering objects in manufacturing lines to locating books on bookshelves. Radio Frequency Identification (RFID) based localization is of great interest in many IoT applications. Synthetic aperture RFID, due to its anti-noise capability and robustness against multipath distortion, is becoming a rising star in the field of localization. Existing systems achieve finer lateral resolution, whereas their radial accuracy is limited by the narrow bandwidth of RFID signal. In this paper, we present a novel synthetic aperture RFID localization method which combines RFID phase based ranging with synthetic aperture technology, to achieve a higher radial accuracy than the existing systems. With only one reader antenna and one 1-dimensional (1D) trajectory, a synthetic array is constructed to get an accurate localization result both in lateral and radial direction. Its core idea is to make full use of the coherence of all multi-frequency phase data and merge them into a unique ranging based likelihood function. To improve the accuracy, the relative phase is leveraged to eliminate phase offsets caused by the reader antenna, and the phase deviation from the angle-of-arrival response is calibrated by pre-processing. Then a weighted enhancement is fully exploited to further improve the localization performance. We evaluate its performance with commercial-off-the-shelf (COTS) RFID devices and the results show that it achieves median accuracy of 3cm in both lateral and radial direction. This novel promising method is suitable for locating tags placed densely in many IoT applications, such as test tubes in hospitals.