Audio adversarial perturbations are designed to remain imperceptible to humans while deceiving automatic speech recognition (ASR) models. However, operating within the audible frequency range makes existing methods partially detectable in practice. In this paper, we present LaserAdv, a laser-based adversarial attack that injects carefully crafted perturbations via laser signals, which are superimposed on speech rather than masking it. This design exploits a well-established property of adversarial examples—the ability to mislead models through minimal, often imperceptible, modifications—while preserving the underlying speech, thereby enabling higher attack efficiency and a longer effective attack range. To mitigate distortion introduced during laser transmission, we propose SAE-TFI, a selective amplitude enhancement method in the time–frequency domain. LaserAdv enables physically realizable attacks that are inaudible and black-box, while supporting targeted and universal attack settings without requiring signal synchronization. Experimental results demonstrate that a single perturbation can cause DeepSpeech, Whisper, and iFlytek to misinterpret any of the 12,260 voice commands as target with accuracy of up to 100%, 92% and 88%, respectively. The maximum effective attack distance reaches 120 m.

With the increasing popularity of voice-centric applications, acoustic eavesdropping attacks pose a significant threat to user privacy. Although smartphones require explicit user permission to access the microphone, such attacks can bypass this restriction by exploiting power consumption data through compromised power supplies, such as USB adapters, public charging stations, and power banks. However, previous attempts can only recognize a limited set of hotwords or digits. To address this limitation, we introduce PowerEar, an acoustic eavesdropping attack that leverages the power side channel to reconstruct any audio reproduced by the built-in loudspeaker of a mobile device with an unconstrained vocabulary. Our approach relies on a combination of signal processing and generative techniques to learn the mapping between power consumption and audio playback, enabling the reconstruction of such audio through spectrogram enhancement. To validate the effectiveness of PowerEar attack, we carry out a comprehensive set of experiments using audio samples from various public personalities. Our results obtained through objective and subjective evaluations clearly demonstrate that PowerEar can successfully recover user speeches from power consumption data in comprehensive realistic settings, including speech utterances of individuals and different devices, mobile operating systems, activities, charging technology, battery and volume levels.

Modern vehicles are widely equipped with in-vehicle audio systems, and drivers routinely play music, podcasts, navigation prompts, phone calls, and various voice services during daily commutes. Due to the enclosed nature of the vehicle cabin and the presence of substantial ambient noise, in-vehicle content is commonly assumed to be inherently private and imperceptible from outside the vehicle. In this work, we show that this assumption does not always hold and reveal a previously unexplored leakage channel for in-vehicle audio. We propose Vehisper, the first systematic study investigating the feasibility of passively eavesdropping on in-vehicle audio from outside a moving vehicle. We discover that, during operation, in-vehicle loudspeakers excite the vehicle's metallic structure, giving rise to observable low-frequency magnetic leakage outside the vehicle, which forms a stealthy side channel that has not been explored in prior work. Building on this insight, we design a compact external sensing device and develop a multi-stage signal processing pipeline to systematically cope with the strong and complex magnetic interference introduced by vehicle motion, as well as spectral distortion and perceptual degradation induced by the leakage channel. We conduct a comprehensive evaluation of Vehisper on 20 commercial vehicles from 10 major manufacturers. Experimental results demonstrate that, under real driving conditions, Vehisper can reliably recover in-vehicle audio, achieving an average word error rate of 7.85%.

Deep learning has significantly advanced Radio Frequency (RF) sensing, leading to extensive research and practical applications in both academia and industry. However, these advancements have also introduced potential privacy and security threats to RF sensing data. In this paper, we present RFInv, the first model inversion attack targeting deep learning classifier-empowered RF sensing systems. RFInv can recover users' private sensing data without knowledge of the RF sensing model's structure, relying solely on the output prediction vector of the deep learning classifier. Consequently, this recovered sensitive data can be exploited for malicious purposes such as identity impersonation and unauthorized device control. To realize the proposed attack, we develop a deep generative adversarial network that integrates an inversion module and a critic module, enabling effective RF data recovery in black-box scenarios. To address the unique challenge of preserving physical consistency in RF data, we incorporate attention mechanisms and deformable convolutions to model their complex temporal and spatial dynamics, ensuring physical consistency. Furthermore, a spectrogram alignment loss is introduced to further enhance reconstruction accuracy. The network is trained using an auxiliary dataset, circumventing the need for access to the target model's training data. We systematically evaluate our proposed attack across multiple datasets for various RF sensing tasks and target models with different network architectures. Extensive experiments demonstrate that RFInv can recover diverse types of RF privacy data with an average Structural Similarity Index Measure (SSIM) of 0.78 and achieves an 86.21% Relative Attack Success Rate (RASR).

Palm vein recognition has gained traction in secure authentication due to its unique, stable, and inherently concealed biometric characteristics. However, the electromagnetic (EM) emissions from subcutaneous vein imaging sensors (SVIS) may unintentionally leak sensitive biometric information, fundamentally challenging the assumed security guarantees. In this paper, we propose VeinPhantom, a novel side-channel attack that reconstructs palm vein patterns from unintended EM emissions of SVIS, ultimately enabling spoofing attacks against biometric authentication systems. To overcome the challenge of low information entropy caused by weak EM signals and complex environmental interference, VeinPhantom first analyzes palm vein information from EM signals, then employs a cascaded enhancement strategy and incorporates a Dynamic Guidance Diffusion framework to progressively reconstruct high-fidelity palm vein patterns. Extensive experiments demonstrate that VeinPhantom achieves an average structure similarity index measure (SSIM) of 0.56 on commercial devices, along with a 56.96% spoofing success rate against state-of-the-art authentication systems. We further discuss potential mitigation strategies to defend against the attack.

Radio Frequency Identification (RFID) technology has emerged as a pervasive modality for human motion sensing in applications such as smart environments and healthcare monitoring. However, the inherent through-wall sensing capability of RFID technology raises critical privacy concerns regarding the unintended leakage of human motion information, a challenge that has not been adequately addressed. To fill this gap, we present a metasurface-based RFID sensing defence (MetaRFence), the first system designed to protect human motion privacy against adversarial through-wall RFID sensing. To this end, we first devise a programmable metasurface comprising 1-bit phase shifters to systematically obfuscate motion-induced signal patterns. Then, we characterize the metasurface's impact on RFID signals across temporal and spectral domains through comprehensive theoretical modeling and empirical investigations. However, our analysis reveals that it is non-trivial to achieve effective signal obfuscation in both domains, primarily due to a fundamental trade-off between increasing temporal signal variation and masking human motion in its spectrum. To overcome this, we judiciously devise a metasurface controlling strategy that jointly optimizes the signal entropy, variance, and spectrum distribution to reach a balance between temporal and spectral motion obfuscation. Our comprehensive experiments demonstrate thatMetaRFencereduces adversarial through-wall motion detection rates to$\leq$6%, decreases the F1-score of human gesture recognition to$\leq$0.11 on average, and amplifies respiration rate estimation errors by 3×, establishing a robust defense mechanism for RFID-based motion privacy protection.

Audio recognition systems have become integral to various applications, including speech-to-text, virtual assistants, and security monitoring, where efficiency and privacy are key concerns. The collaborative recognition system (CRS) partitions and deploys the neural network (NN) across multiple edge devices for cooperative recognition without sharing raw audio data. This distributed approach significantly alleviates the computational burden on the client and ensures data privacy. These advantages make CRS increasingly prevalent in audio recognition applications to improve security, efficiency, and provide timely feedback. However, sharing information during collaboration still poses the risk of exposing original data. To the best of our knowledge, this paper introduces InverCRS, the first inversion attack targeting CRS-empowered audio recognition systems. InverCRS is a generative attack in which the attacker trains a local generative model to take intermediate results as input and output the original audio. Once the generative model is trained, it can perform audio inversion using new intermediate results without the need for further optimization. Furthermore, InverCRS utilizes heuristic algorithms to approximate gradients, making it applicable to both white-box and black-box scenarios. We conduct comprehensive experiments to evaluate the feasibility and efficiency of InverCRS across two real-world audio datasets. The results demonstrate that InverCRS can effectively reconstruct the original audio from various split points within the CRS. Additionally, we investigate two potential defense strategies and provide experimental evaluations of their effectiveness in mitigating this attack.

Unmanned aerial vehicle (UAV) poses a major threat to airspace security and privacy protection. However, existing UAV identification systems face the limitations of feature extraction quality degradation under low signal-to-noise ratio (SNR) conditions, and the average delay of traditional tracking algorithms is difficult to meet the real-time requirements due to high computational complexity. To address these issues, this paper proposes mmRTD, a real-time identification and tracking system for non-cooperative UAV sensing using mmWave radar. To solve the problem of serious feature extraction distortion in low SNR environment, this paper proposes a dynamic period adjustment mechanism driven by spectrum volatility to significantly improve the quality of feature extraction under low SNR conditions. To solve the problem of insufficient real-time performance of traditional tracking methods, this paper proposes a lightweight frame-based tracking framework through the target state detection method to solve the real-time bottleneck caused by high-dimensional signal processing of traditional methods. The experimental results show that the detection accuracy of mmRTD reaches 97.2% within a range of 40 m. In particular, the first detection time of mmRTD is two orders of magnitude higher than that of the traditional scheme, which verifies its potential for real-time identification in complex scenes.

Acoustic eavesdropping against private or confidential spaces is a significant threat in the realm of privacy protection. While the presence of soundproof material would weaken such an attack, current eavesdropping technology may be able to bypass these protections. Fortunately, existing studies either inadequately cover the full spectrum of human speech due to low-frequency responses or rely heavily on the prior knowledge used to train a model. To address these challenges, this paper introduces mmEcho, a new acoustic eavesdropping method that utilizes millimeter-wave signals to sense vibration induced by sound precisely. Through signal processing techniques such as the intra-chirp scheme and phase calibration algorithm, mmEcho achieves micrometer-level vibration extraction without requiring target-related data. To improve the range of eavesdropping attacks while reducing noise, we optimize radar signals by leveraging the widespread availability of multiple antennas on commercial off-the-shelf radars. We comprehensively evaluate the performance of mmEcho in different real-world settings. Experimental results demonstrate that, with the aid of multi-antenna technology, mmEcho can more effectively reconstruct the audio from the target at various distances, directions, sound insulators, reverberating objects, sound levels, and languages. Compared to existing methods, our approach provides better effectiveness without prior knowledge, such as the speech data from the target.

Recent studies have demonstrated the feasibility of eavesdropping on audio via radio frequency signals or videos, which capture physical surface vibrations from surrounding objects. However, these methods are inadequate for intercepting internally transmitted audio through wired media. In this work, we introduce radio-frequency retroreflector attack (RFRA) and bridge this gap by proposing an RFRA-based eavesdropping system,RF-Parrot${}^{\mathbf {2}}$, capable of wirelessly capturing audio signals transmitted through earphone wires. Our system entails embedding a tiny field-effect transistor within the wire to establish a battery-free retroreflector, whose reflective efficiency is correlated with the amplitude of the audio signal. To preserve the details of audio signals, we designed a unique retroreflector using a depletion-mode MOSFET (D-MOSFET). This MOSFET can be triggered by any voltage level present in the audio signals, thus guaranteeing no information loss during activation. However, the D-MOSFET introduces a nonlinear convolution operation on the original audio, resulting in distorted audio eavesdropping. Thus, we devised an engineering solution which utilized a novel convolutional neural network in conjunction with an efficient Parallel WaveGAN vocoder to reconstruct the original audio. Our comprehensive experiments demonstrate a strong similarity between the reconstructed audio and the original, achieving an impressive 95% accuracy in speech command recognition.

Object identification is a pivotal enabling technique for smart home and manufacturing applications. Traditional methodologies for object identification predominantly rely on a singular sensor modality, which inherently limits their ability to furnish a detailed characterization of the target object. Addressing this deficiency, in this paper, we fill this gap by introducing CRFUSION, the first-of-its-kind system that integrates the object RGB image and the radio frequency (RF) signal reflected by the object for fine-grained object identification. CRFUSION leverages the complementary characteristics between visible light and radio frequency modalities to simultaneously determine the category and material of target objects. We design a multifaceted object feature from the RF signal, called the Energy Reflection Factor (ERF), which not only reveals the object texture but complements the image modality for identifying the object category. By integrating the characteristics of radar, we obtain radar feature maps based on the ERF of target objects. Additionally, we have developed a modality fusion network to comprehensively integrate the image and ERF features. We conducted a comprehensive evaluation of CRFUSION using a commercial mmWave radar development board and camera. The results show that CRFUSION achieves a classification accuracy of over 96%, demonstrating its robustness, and potential for application.

Real-time velocity monitoring is pivotal for fault detection of rotating machinery. However, existing methods rely on either troublesome deployments of optical encoders and IMU sensors or various tachometers delivering coarse-grained velocity measurements insufficient for fault detection. To overcome these limitations, we proposeRomeoas the first work to exploit the mmWave radar forrotatingmachinery fault detection by extracting a fine-grained velocity signature. Though mmWave radars should capture instant rotation information with their claimed high sensitivity and sampling rate, direct adoption entails significant efforts for high-precision velocity measurement per radar to handle; particularly, exhausted system calibration and noise interference. To this end, we first develop a phase-velocity model to characterize the relationship between the mmWave signal phase and the fine-grained angular velocity. We then explore the geometric properties of specific positions in the rotation trajectory to precisely calibrate the rotation sensing model, leading to an iterative algorithm for accurate angular velocity measurement. Finally, we propose a simple yet effective fault detection algorithm by extracting a unique velocity signature. Our extensive experiments showRomeoachieves a median error of 0.4$^\circ$/s for fine-grained angular speed measurement, outperforming SOTA solutions with over ×16 angular speed granularity and ×7 measurement precision.

Sound eavesdropping using light has been an area of considerable interest and concern, as it can be achieved over long distances. However, previous work has often lacked stealth (e.g., active emission of laser beams) or been limited in the range of realistic applications (e.g., using direct light from a device’s indicator LED or a hanging light bulb). In this paper, we presentEchoLight, a non-intrusive, passive and long-range sound eavesdropping method that utilizes the extensive reflection of ambient light from vibrating objects to reconstruct sound. We analyze the relationship between reflection light signals and sound signals, particularly in situations where the frequency response of reflective objects and the efficiency of diffuse reflection are suboptimal. Based on this analysis, we have introduced an algorithm based on cGAN to address the issues of nonlinear distortion and spectral absence in the frequency domain of sound. We extensively evaluateEchoLight’s performance in a variety of real-world scenarios. It demonstrates the ability to accurately reconstruct audio from a variety of source distances, attack distances, sound levels, light sources, and reflective materials. Our results reveal that the reconstructed audio exhibits a high degree of similarity to the original audio over 40 meters of attack distance.

Attacks on speech recognition systems often use adversarial or inaudible commands. However, a challenge is that adversarial perturbations typically fall within the audible frequency range, making it difficult to achieve inaudibility. Additionally, the non-linear effects of loudspeakers often cause inaudible commands to become audible at higher power levels. Therefore, minimizing the power requirements of the attack is essential to maintain inaudibility. Another significant obstacle is the conversion of variable-length commands, especially longer ones, into shorter target commands. In this paper, we present UltraAdv, a method for generating long-range adversarial perturbations capable of compromising commands of arbitrary length in closed-box setting. By combining the ultrasonic signal with the normal one, rather than negating it as in DolphinAttack, we significantly improve the energy efficiency, thus enhancing its attack distance. We also propose a dynamically adjustable suppression-interference method based on automatic gain control to address the challenge of mismatched durations between long commands and target commands (length-independent). Experiments demonstrate that using a single perturbation, we achieve impressive success rates of 98.84% and 96.62% and 98.32% across a diverse set of 12,260 speeches on DeepSpeech, iFlytek, and Whisper. The attack range reaches up to 15 m, surpassing DolphinAttack's 5 m range at equivalent power.

Internet of Things (IoT) devices are frequently deployed in highly dynamic environments and need to continuously learn new classes from data streams. Incremental Learning (IL) has gained popularity in IoT as it enables devices to learn new classes efficiently without retraining model entirely. IL involves fine-tuning the model using two sources of data: a small amount of representative samples from the original training dataset and samples from the new classes. However, both data sources are vulnerable to Membership Inference Attack (MIA). Fortunately, the existing MIAs result in poor performance against IL, because they ignore features such as the similarity between old and new models at the old classification layer. This paper presents the first MIA against IL, capable of determining not only whether a sample was used for training/fine-tuning but also distinguishing whether it belongs to the representative dataset or the new classes (unique in IL). Extensive experiments validate the effectiveness of our attack across four real-world datasets. Our attack achieves an average attack success rate of 74.03% in the white-box setting (model structure and parameters are known) and 70.08% in the black-box setting. Importantly, our attack is not sensitive to the IL hyper-parameters (e.g., distillation temperature), confirming its accurate, robust, and practical.

With the increasing popularity of virtual reality applications, there is an increasing demand for more interactive entertainment, learning, social interactions, and other activities on eyewear devices. Recognizing users’ emotion and providing reliable feedback can significantly improve the immersive experience for users. However, previous works in emotion recognition required modifications to existing eyewear devices and the integration of additional sensors, or relied on specialized sensors in expensive commercial-grade eyewear devices, making direct deployment on existing consumer-grade eyewear devices challenging. In this paper, we proposeAccEmo, the first system that analyzes the data from the built-in accelerometer sensor on eyewear devices to accurately recognize human emotion.AccEmofirst employs signal processing technologies to process raw accelerometer data, and then uses a binary classification network to determine whether the accelerometer data is influenced by emotional changes. Subsequently,AccEmoproposes a network architecture based on residual neural network and channel-wise attention mechanism as a universal feature extractor to extract complex features related to human emotions from the accelerometer data. Finally,AccEmouses personalized classifiers to achieve emotion recognition for different users. Extensive performance evaluation ofAccEmoacross diverse users demonstrates an exceptional average accuracy of 94.3%. Additionally, the robustness ofAccEmois validated through evaluations in various scenarios, yielding promising results.

The integration of gaze/eye tracking into virtual and augmented reality devices has unlocked new possibilities, offering a novel human-computer interaction (HCI) modality for on-device extended reality (XR). Emerging applications in XR, such as low-effort user authentication, mental health diagnosis, and foveated rendering, demand real-time eye tracking at high frequencies, a capability that current solutions struggle to deliver. To address this challenge, we present EX-Gaze, an event-based real-time eye tracking system designed for on-device extended reality. EX-Gaze achieves a high tracking frequency of 2KHz, providing decent accuracy and low tracking latency. The exceptional tracking frequency of EX-Gaze is achieved through the use of event cameras, cutting-edge, bio-inspired vision hardware that delivers event-stream output at high temporal resolution. We have developed a lightweight tracking framework that enables real-time pupil region localization and tracking on mobile devices. To effectively leverage the sparse nature of event-streams, we introduce the sparse event-patch representation and the corresponding sparse event patches transformer as key components to reduce computational time. Implemented on Jetson Orin Nano, a low-cost, small-sized mobile device with hybrid GPU and CPU components capable of parallel processing of multiple deep neural networks, EX-Gaze maximizes the computation power of Jetson Orin Nano through sophisticated computation scheduling and offloading between GPUs and CPUs. This enables EX-Gaze to achieve real-time tracking at 2KHz without accumulating latency. Evaluation on public datasets demonstrates that EX-Gaze outperforms other event-based eye tracking methods by striking the best balance between accuracy and efficiency on mobile devices. These results highlight EX-Gaze's potential as a groundbreaking technology to support XR applications that require high-frequency and real-time eye tracking. The code is available at https://github.com/Ningreka/EX-Gaze.

Recent works demonstrated that we can eavesdrop on audio by using radio frequency signals or videos to capture the physical surface vibrations of surrounding objects. They fall short when it comes to intercepting internally transmitted audio through wires. In this work, we first address this gap by proposing a new eavesdropping system, RF-Parrot, that can wirelessly capture the audio signal transmitted in earphone wires. Our system involves embedding a tiny field-effect transistor in the wire to create a battery-free retroreflector, with its reflective efficiency tied to the audio signal’s amplitude. To capture full details of the analog audio signals, we engineered a novel retroreflector using a depletion-mode MOSFET, which can be activated by any voltage of the audio signals, ensuring no information loss. We also developed a theoretical model to demystify the nonlinear transmission of the retroreflector, identifying it as a convolution operation on the audio spectrum. Subsequently, we have designed a novel convolutional neural network-based model to accurately reconstruct the original audio. Our extensive experimental results demonstrate that the reconstructed audio bears a strong resemblance to the original audio, achieving an impressive 95% accuracy in speech command recognition.

Sound eavesdropping using light has been an area of considerable interest and concern, as it can be achieved over long distances. However, previous work has often lacked stealth (e.g., active emission of laser beams) or been limited in the range of realistic applications (e.g., using direct light from a device’s indicator LED or a hanging light bulb). In this paper, we present EchoLight, a non-intrusive, passive and long-range sound eavesdropping method that utilizes the extensive reflection of ambient light from vibrating objects to reconstruct sound. We analyze the relationship between reflection light signals and sound signals, particularly in situations where the frequency response of reflective objects and the efficiency of diffuse reflection are suboptimal. Based on this analysis, we have introduced an algorithm based on cGAN to address the issues of nonlinear distortion and spectral absence in the frequency domain of sound. We extensively evaluate EchoLight’s performance in a variety of real-world scenarios. It demonstrates the ability to accurately reconstruct audio from a variety of source distances, attack distances, sound levels, light intensity, light sources, and reflective materials. Our results reveal that the reconstructed audio exhibits a high degree of similarity to the original audio over 40 meters of attack distance.

With the prevalence of various sensors and smart devices in people’s daily lives, numerous types of information are being sensed. While using such information provides critical and convenient services, we are gradually exposing every piece of our behavior and activities. Researchers are aware of the privacy risks and have been working on preserving privacy while sensing human activities. This survey reviews existing studies on privacy-preserving human activity sensing. We first introduce the sensors and captured private information related to human activities. We then propose a taxonomy to structure the methods for preserving private information from two aspects: individual and collaborative activity sensing. For each of the two aspects, the methods are classified into three levels: signal, algorithm, and system. Finally, we discuss the open challenges and provide future directions.

Video surveillance systems play a crucial role in ensuring public safety and security by capturing and monitoring critical events in various areas. However, traditional surveillance cameras face limitations when it comes to malicious physical damage or obscuring by offenders. To overcome this limitation, we proposem$^{2}$2Vision, which is the first millimeter-wave (mmWave)-based video reconstruction system designed to enhance existing video surveillance cameras.m$^{2}$2Visionutilizes mmWave to sense the profile and motion signature of the target, integrating it with previously acquired visual data about the environment and the target's appearance, thereby facilitating the reconstruction of surveillance video. Specifically, our proposed system incorporates a dual-stage mmWave signal denoising algorithm to efficiently eliminate the noise and multiple-input multiple-output virtual antenna enhanced heatmap generation (MVAE-HG) method to obtain fine-grained mmWave heatmaps responsive to the target's profile and motion information. Moreover, we design the mm2Video generative network that first employs a multi-modal fusion module to fuse the mmWave and pre-acquired visual data, then use a conditional generative adversarial network (cGAN)-based video reconstruction module for surveillance video reconstruction. We conducted comprehensive experiments onm$^{2}$2Visionusing a commercial mmWave radar and four surveillance cameras across various environments, with the participation of seven individuals. Evaluation results show thatm$^{2}$2Visioncan achieve an average structural similarity index measure (SSIM) of 0.93, demonstrating its effectiveness and potential.

NFC tag authentication is crucial for preventing tag misuse. Existing NFC fingerprinting methods use physical-layer signals, which incorporate tag hardware imperfections, for authentication purposes. However, these methods suffer from limitations such as low scalability for a large number of tags or incompatibility with various NFC protocols, hindering practical application. To address these issues, we propose a new NFC fingerprinting scheme called NFChain$^+$. Instead of sticking to the NFC resonant frequency, NFChain$^+$excavates the tag hardware uniqueness from the protocol-agnostic tag response signal using an agile and compatible frequency band of NFC to extract the tag fingerprint from a chain of tag responses over multiple frequencies. This significantly improves fingerprint scalability. However, extracting the desired fingerprint presents two challenges: fingerprint inconsistency under different configurations, and fingerprint variations due to the signal noise in generic readers. To overcome these challenges, we design an effective signal elimination method to remove the effect of device configurations and employ contrastive learning to reduce fingerprint variations for accurate tag authentication. We further cultivate a data augmentation strategy to save the cost of manually collecting fingerprint measurements for training the authentication model. Extensive experiments show that we can achieve as low as 3.4% FRR and 4.1% FAR for over 600 NFC tags.

IEEE INFOCOM 2023 - IEEE Conference on Computer Communications, New York City, NY, USA, 17-20 May 2023

Early detection of fatigue driving is pivotal for the safety of drivers and pedestrians. Traditional approaches mainly employ cameras and wearable sensors to detect fatigue features, which are intrusive to drivers. Recent advances in radio frequency (RF) sensing enable non-intrusive fatigue feature detection from the signal reflected by driver’s body. However, existing RF-based solutions only detect partial or coarse-grained fatigue features, which reduces the detection accuracy. To tackle the above limitations, we propose a mmWave-based fatigue driving detection system, called mmDrive, which can detect multiple fine-grained fatigue features from different body parts. However, achieving accurate detection of various fatigue features during driving encounters practical challenges. Specifically, normal driving activities and driver’s involuntary facial movements inevitably cause interference to fatigue features. Thus, we exploit unique geometric and behavioral characteristics of fatigue features and design effective signal processing methods to remove noises from fatigue-irrelevant activities. Based on the detected fatigue features, we further develop a fatigue determination algorithm to decide the driver’s fatigue state. Extensive experiment results from both simulated and real driving environments show that the average accuracy for detecting nodding and yawning features is about 96%, and the average errors for estimating eye blink, respiration, and heartbeat rates are around 2.21 bpm , 0.54 bpm , and 2.52 bpm , respectively. And the accuracy of the fatigue detection algorithm we proposed reached 97.63%.

Respiration monitoring (RM) is crucial for tracking various health problems. Recently, RFID has been widely employed for lightweight and low-cost RM. However, existing RFID-based RM systems are designed for static environments where no people move around the monitored person. While, in practice, most environments are dynamic with people moving nearby, which introduces dynamic multipath signals and significantly distorts respiration signal, leading to inaccurate RM. In this paper, we aim to realize accurate RFID-based RM in dynamic environments. Our observations show that multipath signals can result in a similar pattern to respiration, leading to apnea mis-detection and inaccurate respiration rate estimation. To address this issue, we first measure respiration anomaly in the signal spectrogram to detect apnea. Second, we successfully remove the multipath effect for respiration rate estimation inspired by intrinsic features of human respiration. Specifically, compared with peoples moving pattern, respiration pattern is regular and periodic. By transforming a normal respiration cycle into a matched filter, real respiration cycles can be extracted from the noisy RFID signal. Respiration rate is then estimated via peak detection. The experiments show that our system achieves the average error of 4.2% and 0.51bpm for apnea detection and respiration rate estimation in dynamic environments, respectively.

Liquid fraud has plagued people with huge health risks. Liquid fraud detection can help to reduce the risk of liquid hazards. However, existing systems that use biochemical tools or radio frequency signals for liquid sensing are either expensive, intrusive, or inconvenient for public use. In this article, we propose HearLiquid, a low-cost and nonintrusive liquid fraud detection system using commodity acoustic devices. Our insight comes from the fact that acoustic impedance of different liquids results in distinct absorption of the acoustic signal across different frequencies when it travels through the liquid. In specific, we extract the liquid’s acoustic absorption and transmission curve (AATC) over multiple frequencies of the acoustic signal for liquid fraud detection. However, accurately measuring the AATC faces multiple challenges. First, due to the hardware diversity and imperfection, different acoustic devices introduce diverse frequency responses, which brings significant deviations to AATCs of the same liquid. Second, different relative positions between acoustic devices and the liquid container result in variations in the AATC, making the detection result inaccurate. To overcome these challenges, we first calibrate the AATC using a dedicated reference AATC to remove the effect of hardware diversity. To bear the variations in AATCs measured from different relative positions, we apply a well-orchestrated data augmentation technique to automatically generate sufficient AATCs for different positions using a small number of collected data. Finally, AATCs are used to train the liquid detection model. We conduct extensive experiments on many important liquid fraud cases and achieve liquid detection accuracy of 92%–97%.

Repetitive activities like breathing and walking account for a large fraction of human activities. Monitoring these activities with sensing technology plays a vital role in numerous applications ranging from health monitoring to manufacturing management. Over the last decade, traditional machine learning approaches and recent end-to-end deep learning paradigms have achieved massive successes in human activity recognition. However, these approaches are mostly scenario dependent and computationally expensive. Moreover, real-world repetitive activities may have varying time intervals between each repetition, which invalidate existing sliding window methods. In this paper, we propose STEM, a Scalable Template Extraction Method for scenario independent monitoring of repetitive activities with varying intervals. Instead of using sliding windows, we detect and locate the appearance of repeating patterns based on the Matrix Profile. Distributional features are then extracted from the identified patterns such that domain knowledge can be avoided. The approach is efficient and robust as shown by the evaluation on three public datasets, in which around 95% of the undesired computation were eliminated with up to 4% accuracy improvement. It is also generic as demonstrated by a use case of respiration rate estimation using wireless signals.

In recent years, blockchain has been broadly applied to industrial Internet of Things (IIoT) due to its features of decentralization, transparency, and immutability. In existing permissioned blockchain based IIoT solutions, transactions submitted by IIoT devices are arbitrarily packed into blocks without considering their waiting times. Hence, there will be a high deviation of the transaction response times, which is known as the lack of fairness. Unfair permissioned blockchain decreases the quality of experience from the perspective of the IIoT devices. Moreover, some transactions can get timeouts if not responded for a long time. In this article, we propose Fair-Pack, the first fairness-based transaction packing algorithm for permissioned blockchain empowered IIoT systems. First, we gain the insight that fairness is positively related to the sum of waiting times of the selected transactions through theoretical analysis. Based on this insight, we transform the fairness problem into the subset sum problem, which is to find a valid subset from a given set with subset sum as large as possible. However, it is time consuming to solve the problem using a brute-force approach because there is an exponential number of subsets for a given set. To this end, we propose a heuristic and a min-heap-based optimal algorithm for different parameter settings. Finally, we analyze the time complexity of Fair-Pack and conduct extensive experiments. The results reveal that Fair-Pack is time-efficient and outperforms the existing algorithms significantly in terms of both fairness and average transaction response time.

This paper studies the problem of tag localization using RFID-augmented robots, which is practically important for promising warehousing applications, e.g., automatic item fetching and misplacement detection. Existing RFID localization systems suffer from one or more of following limitations: requiring specialized devices; only 2D localization is enabled; having blind zone for mobile localization; low scalability. In this paper, we use Commercial Off-The-Shelf (COTS) robot and RFID devices to implement a Mobile RF-robot Localization (MRL) system. Specifically, when the RFID-augmented robot moves along the straight aisle in a warehouse, the reader keeps reading the target tag via two vertically deployed antennas ( Z1 and Z2) and returns the tag phase data with timestamps to the server. We take three points in the phase profile of antenna Z1 and leverage the spatial and temporal changes inherent in this phase triad to construct an equation set. By solving it, we achieve the location of target tag relative to the trajectory of antenna Z1. Based on different phase triads, we can have candidate locations of the target tag with different accuracy. Then, we propose theoretical analysis to quantify the deviation of each localization result. A fine-grained localization result can be achieved by assigning larger weights to the localization results with smaller deviations. Similarly, we can also calculate the relative location of target tag with respect to the trajectory of antenna Z2. Leveraging the geometric relationships among target tag and antenna trajectories, we eventually calculate the location of target tag in 3D space. We perform various experiments to evaluate the performance of the MRL system and results show that the proposed MRL system can achieve high accuracy in both 2D and 3D localization.

Respiration monitoring (RM) is essential for diagnosing and tracking respiratory diseases. Recently, RFID technology has enabled RM in a lightweight and cost-effective way by only attaching the tiny and cheap RFID tag on the monitored person's chest. However, current systems are mostly designed for static environments with no surrounding people's movements. In reality, dynamic environments where people could move nearby the monitored person are quite common. In such environments, respiration signals would be disturbed by the dynamic multipath signals from ambient movements, which may lead to inaccurate RM results. In this paper, we study how to realize robust RFID-based RM in dynamic environments with accurate respiration rate estimation and apnea detection. We find that the dynamic multipath signals can cause not only high-frequency noises but also fake and distorted respiration cycles, which cannot be simply removed by the low-pass filter. Thus, we need a new method to eliminate the effect of multipath signals. Inspired by the intrinsic features of human respiration pattern, we propose to transform the respiration pattern into a matched filter, which can extract the real respiration cycles out of noisy RFID signals. We then estimate the respiration rate by counting the respiration cycles via multi-scale peak detection. For apnea detection, the problem from multipath signals is that the fake respiration cycles can result in the missing detection of apnea when the monitored person stops breathing. To address this issue, we define a new indicator which measures the dominance of respiration components in the signal's spectrum to identify apnea from multipath signals. Experimental results show that our system achieves an average error of 0.5 bpm for respiration rate estimation and a 5.3% error for apnea detection in dynamic environments.

Visitor counting can be attractive to various applications, like business management and marketing investigation. Recently, many studies have employed wireless signals to achieve visitor counting without people's active participation and privacy intrusion. However, existing systems mainly count the overall visitors inside a certain area, which fails to provide the fine-grained information of the coming and leaving visitor flow. Unlike previous studies, this article proposes to count the in-and-out visitors to monitor visiting frequency and population, which can be applied for many indoor places, such as shops and restaurants. Therefore, we present the first WiFi-based in-and-out visitor counting system, Door-Monitor, which obtains the direction (enter or exit) and the number of visitors passing by the door. The WiFi signals enable us to count the visitors in a low-cost and nonintrusive way, and it can tell the exact number of visitors even when multiple persons pass by the door simultaneously. To detect the visitors' passing direction, we show that the patterns in the phase difference series can indicate the entering and exiting passing directions by analyzing the effects of the passing behavior on the signal's phase information. To count the passing visitors, we perform a short time Fourier transformation on the phase difference series to generate the spectrogram, on which the convolutional neural network is applied for building a counting model. The experimental results show that the average accuracies of passing direction detection and visitor counting are 95.2% and 94.5%, respectively.

Human respiration state is an important indicator to reflect health conditions. Recent advances in wireless human sensing have enabled device-free respiration monitoring using narrow-band wireless signals, which, however, fail to map the estimated respiration states to multiple persons. In this paper, we present Multi-Breath, a UWB-based system to achieve separate respiration monitoring for multiple persons. The UWB radar can accurately measure the travelling distance of the signals, which helps to separate the signals affected by different persons and map the detected respiration patterns to the corresponding persons with the location information. However, the radar signal time series of each person are quite noisy due to the multi-path effects caused by the respiration movements of other persons, making it difficult to accurately estimate the respiration state. To overcome this challenge, we propose to transform the UWB radar signal matrices of different persons as separate RGB images to reveal the respiration pattern of each individual. Then, the image processing operations, including image smoothing, edge detection, dilation and erosion, are applied to identify the breathing cycles. Finally, the respiration state, including the respiration rate and the presence of apnea, is estimated via blob detection and calibration. Extensive experiments show that the mean absolute error on respiration rate estimation is 0.3 - 0.6 bpm, and the percentage of missed and false detected apnea is 3% - 7%.

Supply chain management (SCM) is fundamental for gaining financial, environmental and social benefits in the supply chain industry. However, traditional SCM mechanisms usually suffer from a wide scope of issues such as lack of information sharing, long delays for data retrieval, and unreliability in product tracing. Recent advances in blockchain technology show great potential to tackle these issues due to its salient features including immutability, transparency, and decentralization. Although there are some proof-of-concept studies and surveys on blockchain-based SCM from the perspective of logistics, the underlying technical challenges are not clearly identified. In this paper, we provide a comprehensive analysis of potential opportunities, new requirements, and principles of designing blockchain-based SCM systems. We summarize and discuss four crucial technical challenges in terms of scalability, throughput, access control, data retrieval and review the promising solutions. Finally, a case study of designing blockchain-based food traceability system is reported to provide more insights on how to tackle these technical challenges in practice.

This paper studies the important sensory data collection problem in the sensor-augmented RFID systems, which is to quickly and accurately collect sensory data from a predefined set of target tags with the coexistence of unexpected tags. The existing RFID data collection schemes suffer from either low time-efficiency due to tag-collisions or serious data corruption issue due to interference of unexpected tags. To overcome these limitations, we propose the hierarchical-hashing data collection (HDC) protocol, which can not only significantly improve the utilization of RFID wireless communication channel by establishing bijective mapping between k target tags and the first k slots in time frame, but also effectively filter out the serious interference of unexpected tags. Although HDC has attractive advantages, the theoretical analysis reveals that the computation cost involved in it is as huge as O(k2k), where k is normally large in practice. By making some modifications to the basic HDC protocol, we propose the multi-framed hierarchical-hashing data collection (MHDC) protocol to effectively reduce the involved computation complexity. Unlike HDC that only issues a single time frame, MHDC uses multiple time frames to collaboratively collect sensory data from the k target tags. It can be understood as that a big computation task is disintegrated into multiple small pieces and then shared by multiple time frames. As a result, the computation cost involved in MHDC is reduced to O(k2n), where n ≪ k is the expected number of target tags that each time frame handles. Theoretical analysis is given to jointly consider the communication cost and computation cost thereby maximizing the overall time-efficiency of MHDC. Extensive simulation results reveal that the proposed MHDC protocol can correctly collect all sensory data and is always about more than 2× faster than the state-of-the-art RFID sensory data collection protocols.

People counting provides valuable information on population mobility and human dynamics, which plays a critical role for intelligent crowd control and retail management. Recently, people counting has been achieved via radio-frequency signals as human presence can influence the propagation of wireless signals, from which the information of the moving crowd can be extracted. However, most of the existing studies using wireless signals only apply to the scenario when people keep moving all the time. Besides, they require labour-intensive training phase for building the counting model. In the Wi-Count system, we take another approach, which is to count the people passing by the doorway with COTS WiFi devices. It can not only detect the passing direction, but also identify the number of people even when multiple persons pass by concurrently without regulating passing behavior and pre-trained counting model. The passing direction is recognized by modeling the effects of the bi-directional passing behavior on the phase difference of WiFi signals. In addition, the number of passing people is obtained through an enhanced signal separation algorithm for providing precise counting result. Extensive experiments show the average accuracy on passing direction detection and passing people counting are about 95% and 92% respectively.

Recently, non-intrusive respiration monitoring has attracted much attention. Many respiration monitoring systems using the commercial off-the-shelf WiFi devices have been developed. However, these systems mainly have difficulties in the presence of multiple persons. The difficulty generally comes from the separation of the effects of multiple persons' respiration on the received WiFi signals. Another problem is that even though the separation can be feasible with some complicated algorithms, it is still impossible to map the multiple identified respiration states to the corresponding persons. In this paper, we study the problem of multi-person sleeping respiration monitoring and try to address the above challenges. Instead of focusing on developing complicated signal processing algorithms, we take another approach: via the deployment of WiFi transceivers. The key insight comes from the WiFi Fresnel zone model, which indicates that a carefully placed WiFi transceiver may only be affected by the person in a certain location. Furthermore, we consider the sleeping movements of people as well as the sleeping posture change to improve the robustness of the system. Extensive experiments show that we can successfully estimate the respiration rate of multiple persons, with the Mean Absolute Error (MAE) of 0.5 bpm - 1 bpm.

Nowadays, a great number of healthcare data are generated every day from both medical institutions and individuals. Healthcare information exchange (HIE) has been proved to benefit the medical industry remarkably. To store and share such large amount of healthcare data is important while challenging. In this paper, we propose BlocHIE, a Blockchain-based platform for healthcare information exchange. First, we analyze the different requirements for sharing healthcare data from different sources. Based on the analysis, we employ two loosely-coupled Blockchains to handle different kinds of healthcare data. Second, we combine off-chain storage and on-chain verification to satisfy the requirements of both privacy and authenticability. Third, we propose two fairness-based packing algorithms to improve the system throughput and the fairness among users jointly. To demonstrate the practicability and effectiveness of BlocHIE, we implement BlocHIE in a minimal-viable-product way and evaluate the proposed packing algorithms extensively.

Nowadays is the big data era. A large amount of data are generated which can be valuable for business, healthcare, transportation, etc. To promote the dissemination of the valuable data, researchers have been trying to design and develop data sharing platforms. However, the existing platforms fail to address at least one of the three issues: trustworthiness, data heterogeneity, and authenticability. To this end, we propose TSAR, a fully-distributed Trustless data ShARing platform. In detail, we architect TSAR on Blockchain to remove the dependency on reliable third parties, which realizes the trustworthiness. Moreover, we propose a general data schema to represent raw data, which handles the problem of data heterogeneity. Finally, we record the data transaction as well as user-group information on Blockchain to achieve authenticability. To demonstrate the practicability and effectiveness of TSAR, we implement it in a minimal-viable-product fashion and evaluate the performance in terms of throughput and response time.