In-house occupancy detection is vital for smart energy management, resource optimization, and home security. Traditional sensor-based solutions can be inaccurate and invasive. This paper presents a WiFi-based occupancy detection system that utilizes existing WiFi infrastructure and IoT devices. Our method employs a neural network with a shared CNN and a transformer block. Our preliminary evaluation, conducted using 18 unique IoT devices and data collected from 7 different homes over 42 days, demonstrates a detection accuracy of 94.88% with 8.12% false alarms in familiar environments, and 90.79% accuracy with 10.11% false alarms in new settings, significantly outperforming model-based methods.

Detecting the target speaker and enhancing speech with high fidelity has been a long-standing problem, especially in challenging acoustic conditions. To address these problems with minimal user cooperation, we have developed multimodal audioradio speech detection (RadioVAD) and enhancement (RadioSES) systems using mmWave modality. This demo presents how these two systems can be run together in real time to extract target speech and filter any type of ambient noise. Our demo uses an mmWave radar and a microphone attached to a laptop to detect and localize target speakers in the field of view, detect the presence of voice to trigger the microphone, and enhance the noisy speech with a deep learning model running in real-time. Our experiments confirm that an audioradio system can detect and isolate high-fidelity target speech, even with interfering speech and noise; while being privacy preserving and environmentally robust.

Voice interfaces have become one of the most ubiquitous human–computer interaction methods in recent years. Voice activity detection (VAD) is typically the first building block of a complex voice interface, often relying on audio signals. Acoustics-based VAD systems do not perform well in noisy and interference-prone environments. Smart assistants mitigate this problem by using a dictionary-based detection system. However, this approach is limited in its applicability. For instance, users may still need to manually mute and unmute their microphones during online meetings to prevent detection of interfering users, and speech leakage. In order to automate voice detection in challenging environments without these limitations, we propose RadioVAD, a noise and interference-resilient VAD system that uses radio modality, which is already available in various smartphones and home assistants. RadioVAD works by detecting possible human presence in the Field of View of the device, extracting the vocal fold’s vibration signal from the target speaker, and utilizing a time-domain neural network on raw radio signals to detect voice activity. Extensive experiments reveal that RadioVAD can detect voice activity in challenging environments with high accuracy and outperforms audio-based VAD when the audio signal has signal-to-noise ratio below 5 dB. Furthermore, RadioVAD reduces false alarm rate in interference-prone environments by 52%–72%, bringing significant improvements to VAD task. RadioVAD lays the foundation for future voice interfaces utilizing radio modality.

Indoor target detection through motion sensing based on Wi-Fi signals has gained much attention recently. However, most of the existing motion detection approaches can only detect motion in a large coverage area without knowing the distance of the target motion from the transmitter (Tx)/receiver (Rx). Passive positioning techniques can provide the location of a target, which, however, requires high deployment efforts without robust performance. In this article, we present a novel technique for detecting motion in proximity by exploring the physics behind the indoor radio frequency (RF) multipath propagation. We discover that motion in the proximity of the Rx/Tx produces distinct time dispersion over the radio channel at the Rx/Tx side. By exploring two novel metrics and linking them with the distance of the motions to antennas, we are able to precisely distinguish motions in nearby proximity from the motions far away. Extensive experiments in various real-world scenarios demonstrate that the proposed scheme can achieve true positive rates (TPRs) greater than 95% and 99% in distance-based and room-level proximity detection, respectively, while maintaining the corresponding false positive rates (FPRs) less than 5% and 0.5%. The detection delays for a detection distance of 2 m are within 0.6 s, which verifies the responsiveness of the proposed scheme.

Voice interfaces have become an integral part of our lives with the proliferation of smart devices. Today, Internet of Things devices mainly rely on microphones to sense sound. Microphones, however, have fundamental limitations, such as weak source separation, limited range in the presence of acoustic insulation, and being prone to multiple side-channel attacks. In this article, we propose RadioMic, a radio-based sound sensing system to mitigate these issues and enrich sound applications. RadioMic constructs sound based on tiny vibrations on active sources (e.g., a speaker diaphragm) or object surfaces (e.g., paper bag), and can work through walls, even a soundproof one. To convert the extremely weak sound vibration in the radio signals into sound signals, RadioMic introduces radio acoustics, and presents training-free approaches for robust sound detection and high-fidelity sound recovery. It then exploits a neural network to further enhance the recovered sound by expanding the recoverable frequencies and reducing the noises. RadioMic translates massive online audios to synthesized data to train the network and, thus, minimizes the need for radio-frequency (RF) data. We thoroughly evaluate different components of RadioMic under different scenarios using a commodity mmWave radar. The results show RadioMic outperforms the state-of-the-art systems significantly. We believe RadioMic provides new horizons for sound sensing and inspires attractive sensing capabilities of mmWave sensing devices.

Speech enhancement and separation have been a long-standing problem, especially with the recent advances using a single microphone. Although microphones perform well in constrained settings, their performance for speech separation decreases in noisy conditions. In this work, we proposeRadioSES, an audioradio speech enhancement and separation system that overcomes inherent problems in audio-only systems. By fusing a complementary radio modality,RadioSEScan estimate the number of speakers, solve the source association problem, separate and enhance noisy mixture speeches, and improve both intelligibility and perceptual quality. We perform millimeter-wave sensing to detect and localize speakers and introduce an audioradio deep learning framework to fuse the separate radio features with the mixed audio features. Extensive experiments using commercial off-the-shelf devices show thatRadioSESoutperforms a variety of state-of-the-art baselines, with consistent performance gains in different environmental settings. Similar to the audiovisual methods,RadioSESprovides significant performance improvements (e.g. 3 dB gains in SiSDR, when compared with the corresponding audio-only method), along with the benefits of lower computational complexity and better privacy preservation.

Speech enhancement and separation have been a long-standing problem with recent advances using a single microphone. With the help of video modality, improvements have been shown for these tasks. In this work, we explore a multimodal approach using mmWave radio devices, as these devices can measure vocal folds vibration. Thorough data collection and extensive experiments with two different neural networks indicate that radio modality can bring significant improvements in speech enhancement and separation.

Monitoring and identifying gait has recently emerged as a promising solution candidate for unobtrusive human recognition. In order to enable ubiquitous and reliable application, a gait recognition system must be robust to environment changes and easy to use without requiring too much user cooperation and recalibration, while maintaining high accuracy, which is often not satisfied in conventional approaches. In this article, we present$\boldsymbol {GaitCube}$, a high-accuracy gait recognition system with the minimal training requirement using a single commodity millimeter-wave (mmWave) radio. To reduce the training overhead, we proposegait data cube, a novel 3-D joint-feature representation of micro-Doppler and micro-range signatures over time that can comprehensively embody the physical relevant features of one’s gait. With a pipeline of signal processing,$\boldsymbol {GaitCube}$can automatically detect and segment human walking and effectively extract thegait data cubes. We implement and evaluate$\boldsymbol {GaitCube}$through experiments conducted at six different locations in a typical indoor space with ten subjects over a month, resulting in >50000 gait instances. The results show that$\boldsymbol {GaitCube}$achieves an accuracy of 96.1% with a single gait cycle using one receive antenna, and the accuracy increases to 98.3% when combining all the receive antennas. Further, it achieves an average recognition accuracy of 79.1% for testing over different times and unseen locations by using only 2 min of training data collected in a single location, enabling a practical and ubiquitous gait-based identification.

Motion detection based on WiFi signals has gained much attention recently. However, most of the existing approaches can only detect motion in a large coverage area without knowing how far the target motion happens. In this paper, we propose two robust and responsive features in the frequency dimension, which are sensitive to the distance of motion, and establish the connection between the underlying radio propagation properties and the features. Extensive experiments in various environments demonstrate that the proposed proximity detection scheme can achieve true positive rates greater than 90% and 98% in corridor and room scenarios, respectively, while maintaining the corresponding false positive rates less than 5% and 1%. The responsiveness of the proposed scheme is verified by measured detection delays within 1.5 s for a detection distance of 2 m.

With the proliferation of smart devices, voice interfaces have become an integral part of our lives, which typically senses sound by microphones through converting the changes in air pressure into electrical signals. Sound sensing through another modality can enable various sensing applications in the absence of a microphone. In fact, environmental sound creates tiny vibrations on object surfaces, which could be captured by radio signals. In this work, we model the vibration on object surfaces due to sound for mmWave devices. We propose a method for the recovery of sound and conduct experiments with various materials to investigate the feasibility of sound reconstruction. We further evaluate the effect of distance and placement to understand the practical limits on the sound reconstruction. The results show that, by using a commodity off-the-shelf radar, it is possible to capture a significant amount of sound from the environment.