Acoustic sensing manifests great potential in various applications like health monitoring, gesture interface, by utilizing built-in speakers and microphones on smart devices. However, in ongoing research and development, one problem is often overlooked: the same speaker, when used concurrently for sensing and other traditional audio tasks (like playing music), could cause interference in both, making it impractical to use. The strong ultrasonic sensing signals mixed with music would overload the speaker’s mixer. To confront this issue of overloaded signals, current solutions are clipping or down-scaling, both of which affect the music playback quality, sensing range, and accuracy. To address this challenge, we propose CoPlay, a deep learning-based optimization algorithm to cognitively adapt the sensing signal and run in real-time. It can 1) maximize the sensing signal magnitude within the available bandwidth left by the concurrent music to optimize sensing range and accuracy and 2) minimize any consequential frequency distortion that can affect music playback. We design a custom model and test it on common types of sensing signals (sine wave or Frequency Modulated Continuous Wave FMCW) as inputs alongside various agnostic types of concurrent music and speech. First, we micro-benchmark the model performance to show the quality of the generated signals. Secondly, we conducted 2 field studies of downstream acoustic sensing tasks on 2 devices in the real world. A study with 12 users proved that respiration monitoring and gesture recognition using our adapted signal achieve similar accuracy as no-concurrent-music scenarios, whereas baseline methods of clipping or down-scaling manifest worse accuracy. A qualitative study also justifies that CoPlay leaves music untouched, unlike clipping or down-scaling that degrade music quality.

Wireless sensing technologies, leveraging ubiquitous sensors such as acoustics or mmWave, can enable various applications such as human motion and health tracking. However, the recent trend of incorporating deep learning into wireless sensing introduces new challenges, such as the need for extensive training data and poor model generalization. As a remedy, data augmentation is one solution well-explored in other fields such as computer vision; yet they are not directly applicable due to the unique characteristics of wireless signals. Hence, we propose a custom data augmentation framework, WixUp, tailored for wireless human sensing. Our goal is to build a generic data augmentation framework applicable to various tasks, models, data formats, or wireless modalities. Specifically, WixUp achieves this by a custom Gaussian mixture and probability-based transformation, making any data formats capable of an in-depth augmentation at the dense range profile level. Additionally, our mixing-based augmentation enables un-supervised domain adaptation via self-training, allowing model training with no ground truth labels from new users or environments in practice. We extensively evaluated WixUp across four datasets of two sensing modalities (mmWave, acoustics), two model architectures, and three tasks (pose estimation, identification, action recognition). WixUp provides consistent performance improvement (2.79%-84.25%) across these various scenarios and outperforms other data augmentation baselines.

The surging popularity of home assistants and their voice user interface (VUI) have made them an ideal central control hub for smart home devices. However, current form factors heavily rely on VUI, which poses accessibility and usability issues; some latest ones are equipped with additional cameras and displays, which are costly and raise privacy concerns. These concerns jointly motivate Beyond-Voice, a novel high-fidelity acoustic sensing system that allows commodity home assistant devices to track and reconstruct hand poses continuously. It transforms the home assistant into an active sonar system using its existing onboard microphones and speakers. We feed a high-resolution range profile to the deep learning model that can analyze the motions of multiple body parts and predict the 3D positions of 21 finger joints, bringing the granularity for acoustic hand tracking to the next level. It operates across different environments and users without the need for personalized training data. A user study with 11 participants in 3 different environments shows that Beyond-Voice can track joints with an average mean absolute error of 16.47mm without any training data provided by the testing subject.

The surging popularity of home assistants and their voice user interface (VUI) have made them an ideal central control hub for smart home devices. However, current form factors heavily rely on VUI, which poses accessibility and usability issues; some latest ones are equipped with additional cameras and displays, which are costly and raise privacy concerns. These concerns jointly motivate Beyond-Voice, a novel high-fidelity acoustic sensing system that allows commodity home assistant devices to track and reconstruct hand poses continuously. It transforms the device into an active sonar system using its existing onboard microphones and speakers. By feeding a high-resolution range profile to the deep learning model, we can localize 21 finger joints in 3D, bringing the granularity for acoustic hand tracking to the next level. A user study with 11 participants in 3 different environments shows that Beyond-Voice can track joints with an average mean absolute error of 16.47mm for unseen environments and users.

Sensor networks with devices capable of moving could enable applications ranging from precision irrigation to environmental sensing. Using mechanical drones to move sensors, however, severely limits operation time since flight time is limited by the energy density of current battery technology. We explore an alternative, biology-based solution: integrate sensing, computing and communication functionalities onto live flying insects to create a mobile IoT platform. Such an approach takes advantage of these tiny, highly efficient biological insects which are ubiquitous in many outdoor ecosystems, to essentially provide mobility for free. Doing so however requires addressing key technical challenges of power, size, weight and self-localization in order for the insects to perform location-dependent sensing operations as they carry our IoT payload through the environment. We develop and deploy our platform on bumblebees which includes backscatter communication, low-power self-localization hardware, sensors, and a power source. We show that our platform is capable of sensing, backscattering data at 1 kbps when the insects are back at the hive, and localizing itself up to distances of 80 m from the access points, all within a total weight budget of 102 mg.

The vision of tracking small IoT devices runs into the reality of localization technologies --- today it is difficult to continuously track objects through walls in homes and warehouses on a coin cell battery. While Wi-Fi and ultra-wideband radios can provide tracking through walls, they do not last more than a month on small coin and button cell batteries since they consume tens of milliwatts of power. We present the first localization system that consumes microwatts of power at a mobile device and can be localized across multiple rooms in settings like homes and hospitals. To this end, we introduce a multi-band backscatter prototype that operates across 900 MHz, 2.4 and 5 GHz and can extract the backscatter phase information from signals that are below the noise floor. We build sub-centimeter sized prototypes which consume 93 μW and could last five to ten years on button cell batteries. We achieved ranges of up to 60 m away from the AP and accuracies of 2, 12, 50 and 145 cm at 1, 5, 30 and 60 m respectively. To demonstrate the potential of our design, we deploy it in two real-world scenarios: five homes in a metropolitan area and the surgery wing of a hospital in patient pre-op and post-op rooms as well as storage facilities.

We present fingerIO, a novel fine-grained finger tracking solution for around-device interaction. FingerIO does not require instrumenting the finger with sensors and works even in the presence of occlusions between the finger and the device. We achieve this by transforming the device into an active sonar system that transmits inaudible sound signals and tracks the echoes of the finger at its microphones. To achieve sub-centimeter level tracking accuracies, we present an innovative approach that use a modulation technique commonly used in wireless communication called Orthogonal Frequency Division Multiplexing (OFDM). Our evaluation shows that fingerIO can achieve 2-D finger tracking with an average accuracy of 8 mm using the in-built microphones and speaker of a Samsung Galaxy S4. It also tracks subtle finger motion around the device, even when the phone is in the pocket. Finally, we prototype a smart watch form-factor fingerIO device and show that it can extend the interaction space to a 0.5×0.25 m2 region on either side of the device and work even when it is fully occluded from the finger.

We present a contactless solution for detecting sleep apnea events on smartphones. To achieve this, we introduce a novel system that monitors the minute chest and abdomen movements caused by breathing on smartphones. Our system works with the phone away from the subject and can simultaneously identify and track the fine-grained breathing movements from multiple subjects. We do this by transforming the phone into an active sonar system that emits frequency-modulated sound signals and listens to their reflections; our design monitors the minute changes to these reflections to extract the chest movements. Results from a home bedroom environment shows that our design operates efficiently at distances of up to a meter and works even with the subject under a blanket.

We explore the feasibility of achieving computational imaging using Wi-Fi signals. To achieve this, we leverage multi-path propagation that results in wireless signals bouncing off of objects before arriving at the receiver. These reflections effectively light up the objects, which we use to perform imaging. Our algorithms separate the multi-path reflections from different objects into an image. They can also extract depth information where objects in the same direction, but at different distances to the receiver, can be identified. We implement a prototype wireless receiver using USRP-N210s at 2.4 GHz and demonstrate that it can image objects such as leather couches and metallic shapes in line-of-sight and non-line-of-sight scenarios. We also demonstrate proof-of-concept applications including localization of static humans and objects, without the need for tagging them with RF devices. Our results show that we can localize static human subjects and metallic objects with a median accuracy of 26 and 15 cm respectively. Finally, we discuss the limits of our Wi-Fi based approach to imaging.

Near Field Communication (NFC) enables physically proximate devices to communicate over very short ranges in a peer-to-peer manner without incurring complex network configuration overheads. However, adoption of NFC-enabled applications has been stymied by the low levels of penetration of NFC hardware. In this paper, we address the challenge of enabling NFC-like capability on the existing base of mobile phones. To this end, we develop Dhwani, a novel, acoustics-based NFC system that uses the microphone and speakers on mobile phones, thus eliminating the need for any specialized NFC hardware. A key feature of Dhwani is the JamSecure technique, which uses self-jamming coupled with self-interference cancellation at the receiver, to provide an information-theoretically secure communication channel between the devices. Our current implementation of Dhwani achieves data rates of up to 2.4 Kbps, which is sufficient for most existing NFC applications.

We consider the problem of locating devices such as laptops, desktops, smartphones etc. within an office environment, without requiring any special hardware or infrastructure. We consider two widely-studied approaches to indoor localization: (a) those based on Radio Frequency (RF) measurements made by devices with WiFi or cellular interfaces, and (b) those based on Acoustic Ranging (AR) measurements made by devices equipped with a speaker and a microphone. A typical office environment today comprises devices that are amenable to either one or both these approaches to localization. In this paper we ask the question, "How can we combine RF and AR based approaches in synergy to locate a wide range of devices, leveraging the benefits of both approaches?" The key contribution of this paper is Centaur, a system that fuses RF and AR based localization techniques into a single systematic framework that is based on Bayesian inference. Centaur is agnostic to the specific RF or AR technique used, giving users the flexibility of choosing their preferred RF or AR schemes. We also make two additional contributions: making AR more robust in non-line-of-sight settings (EchoBeep) and adapting AR to localize speaker-only devices (DeafBeep). We evaluate the performance of our AR enhancements and that of the Centaur framework through microbenchmarks and deployment in an office environment.