Dialogue models falter in noisy, multi-speaker environments, often producing irrelevant responses and awkward turn-taking. We present AV-Dialog, the first multimodal dialog framework that uses both audio and visual cues to track the target speaker, predict turn-taking, and generate coherent responses. By combining acoustic tokenization with multi-task, multi-stage training on monadic, synthetic, and real audio-visual dialogue datasets, AV-Dialog achieves robust streaming transcription, semantically grounded turn-boundary detection and accurate responses, resulting in a natural conversational flow. Experiments show that AV-Dialog outperforms audio-only models under interference, reducing transcription errors, improving turn-taking prediction, and enhancing human-rated dialogue quality. These results highlight the power of seeing as well as hearing for speaker-aware interaction, paving the way for {spoken} dialogue agents that perform {robustly} in real-world, noisy environments.

Despite their ubiquity, wireless earbuds remain audio-centric due to size and power constraints. We present VueBuds, the first camera-integrated wireless earbuds for egocentric vision, capable of operating within stringent power and form-factor limits. Each VueBud embeds a camera into a Sony WF-1000XM3 to stream visual data over Bluetooth to a host device for on-device vision language model (VLM) processing. We show analytically and empirically that while each camera’s field of view is partially occluded by the face, the combined binocular perspective provides comprehensive forward coverage. By integrating VueBuds with VLMs, we build an end-to-end system for real-time scene understanding, translation, visual reasoning, and text reading; all from low-resolution monochrome cameras drawing under 5mW through on-demand activation. Through online and in-person user studies with 90 participants, we compare VueBuds against smart glasses across 17 visual question-answering tasks, and show that our system achieves response quality on par with Ray-Ban Meta. Our work establishes low-power camera-equipped earbuds as a compelling platform for visual intelligence, bringing rapidly advancing VLM capabilities to one of the most ubiquitous wearable form factors.

Hearables are becoming ubiquitous, yet their sound controls remain blunt: users can either enable global noise suppression or focus on a single target sound. Real-world acoustic scenes, however, contain many simultaneous sources that users may want to adjust independently. We introduce Aurchestra, the first system to provide fine-grained, real-time soundscape control on resource-constrained hearables. Our system has two key components: (1) a dynamic interface that surfaces only active sound classes and (2) a real-time, on-device multi-output extraction network that generates separate streams for each selected class, achieving robust performance for upto 5 overlapping target sounds and letting users mix their environment by customizing per-class volumes, much like an audio engineer mixes tracks. We optimize the model architecture for multiple compute-limited platforms and demonstrate real-time performance on 6 ms streaming audio chunks. Across real-world environments in previously unseen indoor and outdoor scenarios, our system enables expressive per-class sound control and achieves substantial improvements in target-class enhancement and interference suppression. Our results show that the world need not be heard as a single, undifferentiated stream: with Aurchestra, the soundscape becomes truly programmable.

We introduce proactive hearing assistants 1 that automatically identify and separate the wearer's conversation partners, without requiring explicit prompts.Our system operates on egocentric binaural audio and uses the wearer's self-speech as an anchor, leveraging turn-taking behavior and dialogue dynamics to infer conversational partners and suppress others.To enable real-time, on-device operation, we propose a dual-model architecture: a lightweight streaming model runs every 12.5 ms for lowlatency extraction of the conversation partners, while a slower model runs less frequently to capture longer-range conversational dynamics.Results on real-world 2-and 3-speaker conversation test sets, collected with binaural egocentric hardware from 11 participants totaling 6.8 hours, show generalization in identifying and isolating conversational partners in multi-conversation settings.Our work marks a step toward hearing assistants that adapt proactively to conversational dynamics and engagement.

The conventional wisdom has been that designing ultra-compact, battery-constrained wireless hearables with on-device speech AI models is challenging due to the high computational demands of streaming deep learning models. Speech AI models require continuous, real-time audio processing, imposing strict computational and I/O constraints.

In crowded settings, the human brain can focus on speech from a target speaker, given prior knowledge of how they sound. We introduce a novel intelligent hearable system that achieves this capability, enabling target speech hearing to ignore all interfering speech and noise, but the target speaker. A naïve approach is to require a clean speech example to enroll the target speaker. This is however not well aligned with the hearable application domain since obtaining a clean example is challenging in real world scenarios, creating a unique user interface problem. We present the first enrollment interface where the wearer looks at the target speaker for a few seconds to capture a single, short, highly noisy, binaural example of the target speaker. This noisy example is used for enrollment and subsequent speech extraction in the presence of interfering speakers and noise. Our system achieves a signal quality improvement of 7.01 dB using less than 5 seconds of noisy enrollment audio and can process 8 ms of audio chunks in 6.24 ms on an embedded CPU. Our user studies demonstrate generalization to real-world static and mobile speakers in previously unseen indoor and outdoor multipath environments. Finally, our enrollment interface for noisy examples does not cause performance degradation compared to clean examples, while being convenient and user-friendly. Taking a step back, this paper takes an important step towards enhancing the human auditory perception with artificial intelligence.

Despite broad interest in modeling spoken dialogue agents, most approaches are inherently "half-duplex" -restricted to turn-based interaction with responses requiring explicit prompting by the user or implicit tracking of interruption or silence events.Human dialogue, by contrast, is "full-duplex" allowing for rich synchronicity in the form of quick and dynamic turn-taking, overlapping speech, and backchanneling.Technically, the challenge of achieving full-duplex dialogue with LLMs lies in modeling synchrony as pre-trained LLMs do not have a sense of "time".To bridge this gap, we propose Synchronous LLMs for fullduplex spoken dialogue modeling.We design a novel mechanism to integrate time information into Llama3-8b so that they run synchronously with the real-world clock.We also introduce a training recipe that uses 212k hours of synthetic spoken dialogue data generated from text dialogue data to create a model that generates meaningful and natural spoken dialogue, with just 2k hours of real-world spoken dialogue data.Synchronous LLMs outperform state-of-the-art in dialogue meaningfulness while maintaining naturalness.Finally, we demonstrate the model's ability to participate in full-duplex dialogue by simulating interaction between two agents trained on different datasets, while considering Internet-scale latencies of up to 240ms.

Extracting the speech of participants in a conversation amidst interfering speakers and noise presents a challenging problem. In this paper, we introduce the novel task of target conversation extraction, where the goal is to extract the audio of a target conversation based on the speaker embedding of one of its participants. To accomplish this, we propose leveraging temporal patterns inherent in human conversations, particularly turn-taking dynamics, which uniquely characterize speakers engaged in conversation and distinguish them from interfering speakers and noise. Using neural networks, we show the feasibility of our approach on English and Mandarin conversation datasets. In the presence of interfering speakers, our results show an 8.19 dB improvement in signal-to-noise ratio for 2-speaker conversations and a 7.92 dB improvement for 2-4-speaker conversations. Code, dataset available at https://github.com/chentuochao/Target-Conversation-Extraction.

Integrating cameras into wireless smart rings has been challenging due to size and power constraints. We introduce IRIS, the first wireless vision-enabled smart ring system for smart home interactions. Equipped with a camera, Bluetooth radio, inertial measurement unit (IMU), and an onboard battery, IRIS meets the small size, weight, and power (SWaP) requirements for ring devices. IRIS is context-aware, adapting its gesture set to the detected device, and can last for 16-24 hours on a single charge. IRIS leverages the scene semantics to achieve instance-level device recognition. In a study involving 23 participants, IRIS consistently outpaced voice commands, with a higher proportion of participants expressing a preference for IRIS over voice commands regarding toggling a device’s state, granular control, and social acceptability. Our work pushes the boundary of what is possible with ring form-factor devices, addressing system challenges and opening up novel interaction capabilities.

We present the first neural network model to achieve real-time and streaming target sound extraction. To accomplish this, we propose Waveformer, an encoder-decoder architecture with a stack of dilated causal convolution layers as the encoder, and a transformer decoder layer as the decoder. This hybrid architecture uses dilated causal convolutions for processing large receptive fields in a computationally efficient manner, while also leveraging the generalization performance of transformer-based architectures. Our evaluations show as much as 2.2–3.3 dB improvement in SI-SNRi compared to the prior models for this task while having a 1.2–4x smaller model size and a 1.5–2x lower runtime. We provide code, dataset, and audio samples: https://waveformer.cs.washington.edu/.

We present NeuriCam, a novel deep learning-based system to achieve video capture from low-power dual-mode IoT camera systems. Our idea is to design a dual-mode camera system where the first mode is low power (1.1 mW) but only outputs grey-scale, low resolution and noisy video and the second mode consumes much higher power (100 mW) but outputs color and higher resolution images. To reduce total energy consumption, we heavily duty cycle the high power mode to output an image only once every second. The data for this camera system is then wirelessly sent to a nearby plugged-in gateway, where we run our real-time neural network decoder to reconstruct a higher-resolution color video. To achieve this, we introduce an attention feature filter mechanism that assigns different weights to different features, based on the correlation between the feature map and the contents of the input frame at each spatial location. We design a wireless hardware prototype using off-the-shelf cameras and address practical issues including packet loss and perspective mismatch. Our evaluations show that our dual-camera approach reduces energy consumption by 7x compared to existing systems. Further, our model achieves an average greyscale PSNR gain of 3.7 dB over prior single and dual-camera video super-resolution methods and 5.6 dB RGB gain over prior color propagation methods.

We present the first wireless earbud hardware that can perform hearing screening by detecting otoacoustic emissions. The conventional wisdom has been that detecting otoacoustic emissions, which are the faint sounds generated by the cochlea, requires sensitive and expensive acoustic hardware. Thus, medical devices for hearing screening cost thousands of dollars and are inaccessible in low and middle income countries. We show that by designing wireless ear-buds using low-cost acoustic hardware and combining them with wireless sensing algorithms, we can reliably identify otoacoustic emissions and perform hearing screening. Our algorithms combine frequency modulated chirps with wideband pulses emitted from a low-cost speaker to reliably separate otoacoustic emissions from in-ear reflections and echoes. We conducted a clinical study with 50 ears across two healthcare sites. Our study shows that the low-cost earbuds detect hearing loss with 100% sensitivity and 89.7% specificity, which is comparable to the performance of a $8000 medical device. By developing low-cost and open-source wearable technology, our work may help address global health inequities in hearing screening by democratizing these medical devices.

The emergence of water-proof mobile and wearable devices (e.g., Garmin Descent and Apple Watch Ultra) designed for underwater activities like professional scuba diving, opens up opportunities for underwater networking and localization capabilities on these devices. Here, we present the first underwater acoustic positioning system for smart devices. Unlike conventional systems that use floating buoys as anchors at known locations, we design a system where a dive leader can compute the relative positions of all other divers, without any external infrastructure. Our intuition is that in a well-connected network of devices, if we compute the pairwise distances, we can determine the shape of the network topology. By incorporating orientation information about a single diver who is in the visual range of the leader device, we can then estimate the positions of all the remaining divers, even if they are not within sight. We address various practical problems including detecting erroneous distance estimates, addressing rotational and flipping ambiguities as well as designing a distributed timestamp protocol that scales linearly with the number of devices. Our evaluations show that our distributed system running on underwater deployments of 4--5 commodity smart devices can perform pairwise ranging and localization with median errors of 0.5--0.9 m and 0.9--1.6 m. Project page with code: https://underwatergps.cs.washington.edu/

Imagine being able to listen to the birds chirping in a park without hearing the chatter from other hikers, or being able to block out traffic noise on a busy street while still being able to hear emergency sirens and car honks. We introduce semantic hearing, a novel capability for hearable devices that enables them to, in real-time, focus on, or ignore, specific sounds from real-world environments, while also preserving the spatial cues. To achieve this, we make two technical contributions: 1) we present the first neural network that can achieve binaural target sound extraction in the presence of interfering sounds and background noise, and 2) we design a training methodology that allows our system to generalize to real-world use. Results show that our system can operate with 20 sound classes and that our transformer-based network has a runtime of 6.56 ms on a connected smartphone. In-the-wild evaluation with participants in previously unseen indoor and outdoor scenarios shows that our proof-of-concept system can extract the target sounds and generalize to preserve the spatial cues in its binaural output. Project page with code: https://semantichearing.cs.washington.edu

On-device directional hearing requires audio source separation from a given direction while achieving stringent human-imperceptible latency requirements. While neural nets can achieve significantly better performance than traditional beamformers, all existing models fall short of supporting low-latency causal inference on computationally-constrained wearables. We present DeepBeam, a hybrid model that combines traditional beamformers with a custom lightweight neural net. The former reduces the computational burden of the latter and also improves its generalizability, while the latter is designed to further reduce the memory and computational overhead to enable real-time and low-latency operations. Our evaluation shows comparable performance to state-of-the-art causal inference models on synthetic data while achieving a 5x reduction of model size, 4x reduction of computation per second, 5x reduction in processing time and generalizing better to real hardware data. Further, our real-time hybrid model runs in 8 ms on mobile CPUs designed for low-power wearable devices and achieves an end-to-end latency of 17.5 ms.

In this MobiSys demo, we present the demo of our SIGCOMM 2022 paper on underwater messaging system for existing mobile devices like smartphones and smart watches. Our software-only solution leverages audio sensors, i.e., microphones and speakers, ubiquitous in today's devices to enable acoustic underwater communication between mobile devices. To achieve this, we design a communication system that in real-time adapts to differences in frequency responses across mobile devices, changes in multipath and noise levels at different locations and dynamic channel changes due to mobility. Our demo will allow MobiSys attendees to test our system in realtime in a water tank using several demo smart devices in a waterproof pouch. We will also distribute the Android executable of the system via a QR code to attendees who wish to install and try the system on their own smart devices.

In this MobiSys demo we show a low-cost earphone based system that can screen for hearing loss with a cost of $10. Our system is designed to detect otoacoustic emissions (OAE) which are sounds generated when the outer hair cells move in a healthy cochlea and provide information about their function. OAE testing is commonly used as part of universal infant hearing screening protocols in high-income countries [1]. OAE equipment however is expensive hindering early hearing screening in developing countries that bear the disproportionate brunt of disabling hearing loss. Our design sends two pure tones through each of the headphone's ear-buds and records the distortion-product OAEs generated by the cochlea using a microphone. By running algorithms on a smartphone connected to the earphones using its headphone jack, we can detect distortion-product OAEs. Our device has been validated in a clinical study on 201 pediatric ears at oto-laryngology, hearing, and craniofacial clinics, across three different sites and achieved accuracies comparable to a commercial OAE device. In our demo, users will be encouraged to perform this quick test by listening to some tones in their ear through our device to check for the presence of OAEs in their ears.

In this MobiSys demo we present a system to determine fluid properties using the LiDAR sensors present on modern smartphones, as presented in our ACM IMWUT 2022 paper [1]. Traditional methods of measuring properties like viscosity require expensive laboratory equipment or a relatively large amount of fluid. In contrast, our smartphone-based method is accessible, contactless and works with just a single drop of liquid. Our design works by targeting a coherent LiDAR beam from the phone onto the liquid. Using the phone's camera, we capture the characteristic laser speckle pattern that is formed by the interference of light reflecting from light-scattering particles. By correlating the fluctuations in speckle intensity over time, we can characterize the Brownian motion within the liquid which is correlated with its viscosity. Our demo will allow MobiSys attendees to distinguish between liquids of different viscosities, milks of different fat contents, and adulterated milk.

We present ClearBuds, the first hardware and software system that utilizes a neural network to enhance speech streamed from two wireless earbuds. Real-time speech enhancement for wireless earbuds requires high-quality sound separation and background cancellation, operating in real-time and on a mobile phone. Clear-Buds bridges state-of-the-art deep learning for blind audio source separation and in-ear mobile systems by making two key technical contributions: 1) a new wireless earbud design capable of operating as a synchronized, binaural microphone array, and 2) a lightweight dual-channel speech enhancement neural network that runs on a mobile device. Our neural network has a novel cascaded architecture that combines a time-domain conventional neural network with a spectrogram-based frequency masking neural network to reduce the artifacts in the audio output. Results show that our wireless earbuds achieve a synchronization error less than 64 μs and our network has a runtime of 21.4 ms on an accompanying mobile phone. In-the-wild evaluation with eight users in previously unseen indoor and outdoor multipath scenarios demonstrates that our neural network generalizes to learn both spatial and acoustic cues to perform noise suppression and background speech removal. In a user-study with 37 participants who spent over 15.4 hours rating 1041 audio samples collected in-the-wild, our system achieves improved mean opinion score and background noise suppression.

We present ClearBuds, the first end-to-end hardware and software system that utilizes a neural network to enhance speech streamed from two wireless earbuds. Real-time speech enhancement for wireless earbuds requires high-quality sound separation and background cancellation, operating in real-time and on a mobile phone. Clear-Buds bridges state-of-the-art deep learning for blind audio source separation and in-ear mobile systems by making two key technical contributions: 1) a new wireless earbud design capable of operating as a synchronized, binaural microphone array, and 2) a lightweight dual-channel speech enhancement neural network that runs on a mobile device. Our demo will allow MobiSys attendees wear our earbuds, and experience noise suppression as they talk in a noisy environment. Companion video can be accessed using the link below:

Since its inception, underwater digital acoustic communication has required custom hardware that neither has the economies of scale nor is pervasive. We present the first acoustic system that brings underwater messaging capabilities to existing mobile devices like smartphones and smart watches. Our software-only solution leverages audio sensors, i.e., microphones and speakers, ubiquitous in today's devices to enable acoustic underwater communication between mobile devices. To achieve this, we design a communication system that in real-time adapts to differences in frequency responses across mobile devices, changes in multipath and noise levels at different locations and dynamic channel changes due to mobility. We evaluate our system in six different real-world underwater environments with depths of 2--15 m in the presence of boats, ships and people fishing and kayaking. Our results show that our system can in real-time adapt its frequency band and achieve bit rates of 100 bps to 1.8 kbps and a range of 30 m. By using a lower bit rate of 10--20 bps, we can further increase the range to 100 m. As smartphones and watches are increasingly being used in underwater scenarios, our software-based approach has the potential to make underwater messaging capabilities widely available to anyone with a mobile device.

We present the first system that can airdrop wireless sensors from small drones and live insects. In addition to the challenges of achieving low-power consumption and long-range communication, airdropping wireless sensors is difficult because it requires the sensor to survive the impact when dropped in mid-air. Our design takes inspiration from nature: small insects like ants can fall from tall buildings and survive because of their tiny mass and size. Inspired by this, we design insect-scale wireless sensors that come fully integrated with an onboard power supply and a lightweight mechanical actuator to detach from the aerial platform. Our system introduces a first-of-its-kind 37 mg mechanical release mechanism to drop the sensor during flight, using only 450 μJ of energy as well as a wireless communication link that can transmit sensor data at 33 kbps up to 1 km. Once deployed, our 98 mg wireless sensor can run for 1.3-2.5 years when transmitting 10-50 packets per hour on a 68 mg battery. We demonstrate attachment to a small 28 mm wide drone and a moth (Manduca sexta) and show that our insect-scale sensors flutter as they fall, suffering no damage on impact onto a tile floor from heights of 22 m.

Recent years have seen interest in device tracking and localization using acoustic signals. State-of-the-art acoustic motion tracking systems however do not achieve millimeter accuracy and require large separation between microphones and speakers, and as a result, do not meet the requirements for many VR/AR applications. Further, tracking multiple concurrent acoustic transmissions from VR devices today requires sacrificing accuracy or frame rate. We present MilliSonic, a novel system that pushes the limits of acoustic based motion tracking. Our core contribution is a novel localization algorithm that can provably achieve sub-millimeter 1D tracking accuracy in the presence of multipath, while using only a single beacon with a small 4-microphone array.Further, MilliSonic enables concurrent tracking of up to four smartphones without reducing frame rate or accuracy. Our evaluation shows that MilliSonic achieves 0.7mm median 1D accuracy and a 2.6mm median 3D accuracy for smartphones, which is 5x more accurate than state-of-the-art systems. MilliSonic enables two previously infeasible interaction applications: a) 3D tracking of VR headsets using the smartphone as a beacon and b) fine-grained 3D tracking for the Google Cardboard VR system using a small microphone array.

Wireless protocol design for IoT networks is an active area of research. We demonstrate tinySDR which is a low-power software-defined radio platform tailored to the needs of IoT endpoints. TinySDR is a standalone, fully programmable software-defined radio platform which has the requirements of IoT protocols. We present the physical layer implementations of BLE beacon and LoRa protocols to demonstrate the capabilities of tinySDR.

Sensor networks with devices capable of moving could enable applications ranging from precision irrigation to environmental sensing. Using mechanical drones to move sensors, however, severely limits operation time since flight time is limited by the energy density of current battery technology. We explore an alternative, biology-based solution: integrate sensing, computing and communication functionalities onto live flying insects to create a mobile IoT platform. Such an approach takes advantage of these tiny, highly efficient biological insects which are ubiquitous in many outdoor ecosystems, to essentially provide mobility for free. Doing so however requires addressing key technical challenges of power, size, weight and self-localization in order for the insects to perform location-dependent sensing operations as they carry our IoT payload through the environment. We develop and deploy our platform on bumblebees which includes backscatter communication, low-power self-localization hardware, sensors, and a power source. We show that our platform is capable of sensing, backscattering data at 1 kbps when the insects are back at the hive, and localizing itself up to distances of 80 m from the access points, all within a total weight budget of 102 mg.

White noise machines are among the most popular devices to facilitate infant sleep. We introduce the first contactless system that uses white noise to achieve motion and respiratory monitoring in infants. Our system is designed for smart speakers that can monitor an infant's sleep using white noise. The key enabler underlying our system is a set of novel algorithms that can extract the minute infant breathing motion as well as position information from white noise which is random in both the time and frequency domain. We describe the design and implementation of our system, and present experiments with a life-like infant simulator as well as a clinical study at the neonatal intensive care unit with five new-born infants. Our study demonstrates that the respiratory rate computed by our system is highly correlated with the ground truth with a correlation coefficient of 0.938.

In this poster accompanying the MobiCom 2019 paper, we describes how to enable infant monitoring capability using white noise. Specifically, we design a set of novel algorithms that can extract the minute infant breathing motion as well as position information from white noise which is random in both the time and frequency domain. Our study demonstrates that the system achieves high correlation between the measured respiratory rate and the ground truth.

To date, insect scale aerial robots have required wire tethers for providing power due to the challenges of integrating the required high-voltage power electronics within their severely constrained weight budgets. In this paper we present a significant milestone in the achievement of flight autonomy: the first wireless liftoff of a 190 mg aerial vehicle. Our robot is remotely powered using a 976 nm laser and integrates a complete power electronics package weighing a total of 104 mg, using commercially available components and fabricated using a fast-turnaround laser based circuit fabrication technique. The onboard electronics include a lightweight boost converter capable of producing high voltage bias and drive signals of over 200 V at up to 170 Hz and regulated by a microcontroller performing feedback control. We present our system design and analysis, detailed description of our fabrication method, and results from flight experiments.

As connected devices continue to decrease in size, we explore the idea of leveraging everyday surfaces such as tabletops and walls to augment the wireless capabilities of devices. Specifically, we introduce Surface MIMO, a technique that enables MIMO communication between small devices via surfaces coated with conductive paint or covered with conductive cloth. These surfaces act as an additional spatial path that enables MIMO capabilities without increasing the physical size of the devices themselves. We provide an extensive characterization of these surfaces that reveal their effect on the propagation of EM waves. Our evaluation shows that we can enable additional spatial streams using the conductive surface and achieve average throughput gains of 2.6-3x for small devices. Finally, we also leverage the wideband characteristics of these conductive surfaces to demonstrate the first Gbps surface communication system that can directly transfer bits through the surface at up to 1.3Gbps.

Wireless video streaming has traditionally been considered an extremely power-hungry operation. Existing approaches optimize the camera and communication modules individually to minimize their power consumption. However, designing a video streaming device requires power-consuming hardware components and video CODEC algorithms which makes battery-free video streaming currently infeasible. Existing RF-powered wireless camera prototypes require extensive duty-cycling on the order of tens of minutes, to capture, process and communicate a single frame. Self-powered cameras can capture an image once every few seconds, but do not have the capability to stream video wirelessly.

The vision of tracking small IoT devices runs into the reality of localization technologies --- today it is difficult to continuously track objects through walls in homes and warehouses on a coin cell battery. While Wi-Fi and ultra-wideband radios can provide tracking through walls, they do not last more than a month on small coin and button cell batteries since they consume tens of milliwatts of power. We present the first localization system that consumes microwatts of power at a mobile device and can be localized across multiple rooms in settings like homes and hospitals. To this end, we introduce a multi-band backscatter prototype that operates across 900 MHz, 2.4 and 5 GHz and can extract the backscatter phase information from signals that are below the noise floor. We build sub-centimeter sized prototypes which consume 93 μW and could last five to ten years on button cell batteries. We achieved ranges of up to 60 m away from the AP and accuracies of 2, 12, 50 and 145 cm at 1, 5, 30 and 60 m respectively. To demonstrate the potential of our design, we deploy it in two real-world scenarios: five homes in a metropolitan area and the surgery wing of a hospital in patient pre-op and post-op rooms as well as storage facilities.

We present the first wireless physical analytics system for 3D printed objects using commonly available conductive plastic filaments. Our design can enable various data capture and wireless physical analytics capabilities for 3D printed objects, without the need for electronics. To achieve this goal, we make three key contributions: (1) demonstrate room scale backscatter communication and sensing using conductive plastic filaments, (2) introduce the first backscatter designs that detect a variety of bi-directional motions and support linear and rotational movements, and (3) enable data capture and storage for later retrieval when outside the range of the wireless coverage, using a ratchet and gear system. We validate our approach by wirelessly detecting the opening and closing of a pill bottle, capturing the joint angles of a 3D printed e-NABLE prosthetic hand, and an insulin pen that can store information to track its use outside the range of a wireless receiver.

Being in academia provides a unique opportunity to explore one's curiosity and build cool systems that advance human understanding of engineering and science. System researchers however also have a responsibility to build systems that can impact practice in the near-term and perhaps, even create a brand new industry. In my talk, I will focus on my efforts so far navigating this chasm between curiosity and impact driven research. Specifically, I will share two research themes I have been working on as an assistant professor at UW CSE, where what initially began as curiosity-driven projects were transformed by the urge to make immediate practical impact as well as be unique. In particular, I will first talk about our journey going from a science-driven project on ambient backscatter (Sigcomm'13) to building wireless backscatter systems that work reliably and address a key pain-point in the industry. Next, I will talk about how we shifted gears from working on wireless gesture recognition (WiSee, Mobicom'13) to addressing a medical need of millions of people in the United States that go undiagnosed from sleep apnea (ApneaApp, Mobisys'15) and our experience licensing our technology to a multi-national medical corporation. Finally, I will share my thoughts on how our research community can help us better navigate this chasm.

This paper enables data storage and interaction with smart fabric, without the need for onboard electronics or batteries. To do this, we present the first smart fabric design that harnesses the ferromagnetic properties of conductive thread. Specifically, we manipulate the polarity of magnetized fabric and encode different forms of data including 2D images and bit strings. These bits can be read by swiping a commodity smartphone across the fabric, using its inbuilt magnetometer. Our results show that magnetized fabric retains its data even after washing, drying and ironing. Using a glove made of magnetized fabric, we can also perform six gestures in front of a smartphone, with a classification accuracy of 90.1%. Finally, using magnetized thread, we create fashion accessories like necklaces, ties, wristbands and belts with data storage capabilities as well as enable authentication applications.

Our goal is to 3D print wireless sensors, input widgets and objects that can communicate with smartphones and other Wi-Fi devices, without the need for batteries or electronics. To this end, we present a novel toolkit for wireless connectivity that can be integrated with 3D digital models and fabricated using commodity desktop 3D printers and commercially available plastic filament materials. Specifically, we introduce the first computational designs that 1) send data to commercial RF receivers including Wi-Fi, enabling 3D printed wireless sensors and input widgets, and 2) embed data within objects using magnetic fields and decode the data using magnetometers on commodity smartphones. To demonstrate the potential of our techniques, we design the first fully 3D printed wireless sensors including a weight scale, flow sensor and anemometer that can transmit sensor data. Furthermore, we 3D print eyeglass frames, armbands as well as artistic models with embedded magnetic data. Finally, we present various 3D printed application prototypes including buttons, smart sliders and physical knobs that wirelessly control music volume and lights as well as smart bottles that can sense liquid flow and send data to nearby RF devices, without batteries or electronics.

We present fingerIO, a novel fine-grained finger tracking solution for around-device interaction. FingerIO does not require instrumenting the finger with sensors and works even in the presence of occlusions between the finger and the device. We achieve this by transforming the device into an active sonar system that transmits inaudible sound signals and tracks the echoes of the finger at its microphones. To achieve sub-centimeter level tracking accuracies, we present an innovative approach that use a modulation technique commonly used in wireless communication called Orthogonal Frequency Division Multiplexing (OFDM). Our evaluation shows that fingerIO can achieve 2-D finger tracking with an average accuracy of 8 mm using the in-built microphones and speaker of a Samsung Galaxy S4. It also tracks subtle finger motion around the device, even when the phone is in the pocket. Finally, we prototype a smart watch form-factor fingerIO device and show that it can extend the interaction space to a 0.5×0.25 m2 region on either side of the device and work even when it is fully occluded from the finger.

We show for the first time that commodity devices can be used to generate wireless data transmissions that are confined to the human body. Specifically, we show that commodity input devices such as fingerprint sensors and touchpads can be used to transmit information to only wireless receivers that are in contact with the body. We characterize the propagation of the resulting transmissions across the whole body and run experiments with ten subjects to demonstrate that our approach generalizes across different body types and postures. We also evaluate our communication system in the presence of interference from other wearable devices such as smartwatches and nearby metallic surfaces. Finally, by modulating the operations of these input devices, we demonstrate bit rates of up to 50 bits per second over the human body.

In this poster, we show for the first time that commodity devices can be used to generate wireless data transmissions that are confined to the human body. Specifically, we show that commodity input devices such as fingerprint sensors and touchpads can be used to transmit information to only wireless receivers that are in contact with the body.

We introduce inter-technology backscatter, a novel approach that transforms wireless transmissions from one technology to another, on the air. Specifically, we show for the first time that Bluetooth transmissions can be used to create Wi-Fi and ZigBee-compatible signals using backscatter communication. Since Bluetooth, Wi-Fi and ZigBee radios are widely available, this approach enables a backscatter design that works using only commodity devices.

We present the first power over Wi-Fi system that delivers power to low-power sensors and devices and works with existing Wi-Fi chipsets. Specifically, we show that a ubiquitous part of wireless communication infrastructure, the Wi-Fi router, can provide far field wireless power without significantly compromising the network's communication performance. Building on our design, we prototype battery-free temperature and camera sensors that we power with Wi-Fi at ranges of 20 and 17 feet respectively. We also demonstrate the ability to wirelessly trickle-charge nickel---metal hydride and lithium-ion coin-cell batteries at distances of up to 28 feet. We deploy our system in six homes in a metropolitan area and show that it can successfully deliver power via Wi-Fi under real-world network conditions without significantly degrading network performance.

We present a contactless solution for detecting sleep apnea events on smartphones. To achieve this, we introduce a novel system that monitors the minute chest and abdomen movements caused by breathing on smartphones. Our system works with the phone away from the subject and can simultaneously identify and track the fine-grained breathing movements from multiple subjects. We do this by transforming the phone into an active sonar system that emits frequency-modulated sound signals and listens to their reflections; our design monitors the minute changes to these reflections to extract the chest movements. Results from a home bedroom environment shows that our design operates efficiently at distances of up to a meter and works even with the subject under a blanket.

We introduce the first forearm-based EMG input system that can recognize fine-grained thumb gestures, including left swipes, right swipes, taps, long presses, and more complex thumb motions. EMG signals for thumb motions sensed from the forearm are quite weak and require significant training data to classify. We therefore also introduce a novel approach for minimally-intrusive collection of labeled training data for always-available input devices. Our dual-observable input approach is based on the insight that interaction observed by multiple devices allows recognition by a primary device (e.g., phone recognition of a left swipe gesture) to create labeled training examples for another (e.g., forearm-based EMG data labeled as a left swipe). We implement a wearable prototype with dry EMG electrodes, train with labeled demonstrations from participants using their own phones, and show that our prototype can recognize common fine-grained thumb gestures and user-defined complex gestures.

There has been recent interest in designing systems that use the tongue as an input interface. Prior work however either require surgical procedures or in-mouth sensor placements. In this paper, we introduce TongueSee, a non-intrusive tongue machine interface that can recognize a rich set of tongue gestures using electromyography (EMG) signals from the surface of the skin. We demonstrate the feasibility and robustness of TongueSee with experimental studies to classify six tongue gestures across eight participants. TongueSee achieves a classification accuracy of 94.17% and a false positive probability of 0.000358 per second using three-protrusion preamble design.

This paper introduces the first design that enables full-duplex communication on battery-free backscatter devices. Specifically, it gives receivers a way to provide low-rate feedback to the transmitter on the same frequency as that of the backscatter transmissions, using neither multiple antennas nor power-consuming cancellation hardware. Our design achieves this goal using only fully-passive analog components that consume near-zero power. We integrate our design with the backscatter network stack and demonstrate that it can minimize energy wastes that occur due to collisions and also correct for errors and changes in channel conditions at a granularity smaller than that of a packet. To show the feasibility of our design, we build a hardware prototype using off-the-shelf analog components. Our evaluation shows that our design cancels the self-interference down to the noise floor, while consuming only 0.25 μW and 0.54 μW of transmit and receive power, respectively.

We explore the feasibility of achieving computational imaging using Wi-Fi signals. To achieve this, we leverage multi-path propagation that results in wireless signals bouncing off of objects before arriving at the receiver. These reflections effectively light up the objects, which we use to perform imaging. Our algorithms separate the multi-path reflections from different objects into an image. They can also extract depth information where objects in the same direction, but at different distances to the receiver, can be identified. We implement a prototype wireless receiver using USRP-N210s at 2.4 GHz and demonstrate that it can image objects such as leather couches and metallic shapes in line-of-sight and non-line-of-sight scenarios. We also demonstrate proof-of-concept applications including localization of static humans and objects, without the need for tagging them with RF devices. Our results show that we can localize static human subjects and metallic objects with a median accuracy of 26 and 15 cm respectively. Finally, we discuss the limits of our Wi-Fi based approach to imaging.

RF-powered computers are small devices that compute and communicate using only the power that they harvest from RF signals. While existing technologies have harvested power from ambient RF sources (e.g., TV broadcasts), they require a dedicated gateway (like an RFID reader) for Internet connectivity. We present Wi-Fi Backscatter, a novel communication system that bridges RF-powered devices with the Internet. Specifically, we show that it is possible to reuse existing Wi-Fi infrastructure to provide Internet connectivity to RF-powered devices. To show Wi-Fi Backscatter's feasibility, we build a hardware prototype and demonstrate the first communication link between an RF-powered device and commodity Wi-Fi devices. We use off-the-shelf Wi-Fi devices including Intel Wi-Fi cards, Linksys Routers, and our organization's Wi-Fi infrastructure, and achieve communication rates of up to 1 kbps and ranges of up to 2.1 meters. We believe that this new capability can pave the way for the rapid deployment and adoption of RF-powered devices and achieve ubiquitous connectivity via nearby mobile devices that are Wi-Fi enabled.

Communication primitives such as coding and multiple antenna processing have provided significant benefits for traditional wireless systems. Existing designs, however, consume significant power and computational resources, and hence cannot be run on low complexity, power constrained backscatter devices. This paper makes two main contributions: (1) we introduce the first multi-antenna cancellation design that operates on backscatter devices while retaining a small form factor and power footprint, (2) we introduce a novel coding mechanism that enables long range communication as well as concurrent transmissions and can be decoded on backscatter devices. We build hardware prototypes of the above designs that can be powered solely using harvested energy from TV and solar sources. The results show that our designs provide benefits for both RFID and ambient backscatter systems: they enable RFID tags to communicate directly with each other at distances of tens of meters and through multiple walls. They also increase the communication rate and range achieved by ambient backscatter systems by 100X and 40X respectively. We believe that this paper represents a substantial leap in the capabilities of backscatter communication.

In multiuser MIMO (MU-MIMO) networks, the optimal bit rate of a user is highly dynamic and changes from one packet to the next. This breaks traditional bit rate adaptation algorithms, which rely on recent history to predict the best bit rate for the next packet. To address this problem, we introduce TurboRate, a rate adaptation scheme for MU-MIMO LANs. TurboRate shows that clients in an MU-MIMO LAN can adapt their bit rate on a per-packet basis if each client learns two variables: Its SNR when it transmits alone to the access point, and the direction along which its signal is received at the AP. TurboRate also shows that each client can compute these two variables passively without exchanging control frames with the access point. A TurboRate client then annotates its packets with these variables to enable other clients to pick the optimal bit rate and transmit concurrently to the AP. A prototype implementation in USRP-N200 shows that traditional rate adaptation does not deliver the gains of MU-MIMO WLANs, and can interact negatively with MU-MIMO, leading to low throughput. In contrast, enabling MU-MIMO with TurboRate provides a mean throughput gain of 1.7× and 2.3×, for 2-antenna and 3-antenna APs, respectively.

Recent years have witnessed increasing interest in interference alignment which has been demonstrated to deliver gains for wireless networks both analytically and empirically. Typically, interference alignment is achieved by having a MIMO sender precode its transmission to align it at the receiver. In this paper, we show, for the first time, that interference alignment can be achieved via motion, and works even for single-antenna transmitters. Specifically, this alignment can be achieved purely by sliding the receiver's antenna. Interestingly, the amount of antenna displacement is of the order of one inch which makes it practical to incorporate into recent sliding antennas available on the market. We implemented our design on USRPs and demonstrated that it can deliver 1.98× throughput gains over 802.11n in networks with both single-antenna and multi- antenna nodes.

This paper presents WiSee, a novel gesture recognition system that leverages wireless signals (e.g., Wi-Fi) to enable whole-home sensing and recognition of human gestures. Since wireless signals do not require line-of-sight and can traverse through walls, WiSee can enable whole-home gesture recognition using few wireless sources. Further, it achieves this goal without requiring instrumentation of the human body with sensing devices. We implement a proof-of-concept prototype of WiSee using USRP-N210s and evaluate it in both an office environment and a two- bedroom apartment. Our results show that WiSee can identify and classify a set of nine gestures with an average accuracy of 94%.

Recent years have seen major innovations in cross-layer wireless designs. Despite demonstrating significant throughput gains, hardly any of these technologies have made it into real networks. Deploying cross-layer innovations requires adoption from Wi-Fi chip manufacturers. Yet, manufacturers hesitate to undertake major investments without a better understanding of how these designs interact with real networks and applications.

We present the design of a communication system that enables two devices to communicate using ambient RF as the only source of power. Our approach leverages existing TV and cellular transmissions to eliminate the need for wires and batteries, thus enabling ubiquitous communication where devices can communicate among themselves at unprecedented scales and in locations that were previously inaccessible.

This demo presents WiSee, a novel human-computer interaction system that leverages wireless networks (e.g., Wi-Fi), to enable sensing and recognition of human gestures and motion. Since wire- less signals do not require line-of-sight and can traverse through walls, WiSee enables novel human-computer interfaces for remote device control and building automation. Further, it achieves this goal without requiring instrumentation of the human body with sensing devices. We integrate WiSee with applications and demonstrate how WiSee enables users to use gestures and control applications including music players and gaming systems. Specifically, our demo will allow SIGCOMM attendees to control a music player and a lighting control device using gestures.

In multiuser MIMO (MU-MIMO) networks, the optimal bit rate of a user is highly dynamic and changes from one packet to the next. This breaks traditional bit rate adaptation algorithms, which rely on recent history to predict the best bit rate for the next packet. To address this problem, we introduce TurboRate, a rate adaptation scheme for MU-MIMO LANs. TurboRate shows that clients in a MU-MIMO LAN can adapt their bit rate on a per-packet basis if each client learns two variables: its SNR when it transmits alone to the access point, and the direction along which its signal is received at the AP. TurboRate also shows that each client can compute these two variables passively without exchanging control frames with the access point. A TurboRate client then annotates its packets with these variables to enable other clients to pick the optimal bit rate and transmit concurrently to the AP. A prototype implementation in USRP-N200 shows that traditional rate adaptation does not deliver the gains of MU-MIMO WLANs, and can interact negatively with MU-MIMO, leading to low throughput. In contrast, enabling MU-MIMO with TurboRate provides a mean throughput gain of 1.7x and 2.3x, for 2-antenna and 3-antenna APs respectively.

There is a growing interest in physical layer security. Recent work has demonstrated that wireless devices can generate a shared secret key by exploiting variations in their channel. The rate at which the secret bits are generated, however, depends heavily on how fast the channel changes. As a result, existing schemes have a low secrecy rate and are mainly applicable to mobile environments. In contrast, this paper presents a new physical-layer approach to secret key generation that is both fast and independent of channel variations. Our approach makes a receiver jam the signal in a manner that still allows it to decode the data, yet prevents other nodes from decoding. Results from a testbed implementation show that our method is significantly faster and more accurate than state of the art physical-layer secret key generation protocols. Specifically, while past work generates up to 44 secret bits/s with a 4% bit disagreement between the two devices, our design has a secrecy rate of 3-18 Kb/s with 0% bit disagreement.

Recent studies show that high-power cross-technology interference is becoming a major problem in today's 802.11 networks. Devices like baby monitors and cordless phones can cause a wireless LAN to lose connectivity. The existing approach for dealing with such high-power interferers makes the 802.11 network switch to a different channel; yet the ISM band is becoming increasingly crowded with diverse technologies, and hence many 802.11 access points may not find an interference-free channel.

Wireless communication has become an intrinsic part of modern implantable medical devices (IMDs). Recent work, however, has demonstrated that wireless connectivity can be exploited to compromise the confidentiality of IMDs' transmitted data or to send unauthorized commands to IMDs---even commands that cause the device to deliver an electric shock to the patient. The key challenge in addressing these attacks stems from the difficulty of modifying or replacing already-implanted IMDs. Thus, in this paper, we explore the feasibility of protecting an implantable device from such attacks without modifying the device itself. We present a physical-layer solution that delegates the security of an IMD to a personal base station called the shield. The shield uses a novel radio design that can act as a jammer-cum-receiver. This design allows it to jam the IMD's messages, preventing others from decoding them while being able to decode them itself. It also allows the shield to jam unauthorized commands---even those that try to alter the shield's own transmissions. We implement our design in a software radio and evaluate it with commercial IMDs. We find that it effectively provides confidentiality for private data and protects the IMD from unauthorized commands.

This paper presents the design and implementation of 802.11n+, a fully distributed random access protocol for MIMO networks. 802.11n+ allows nodes that differ in the number of antennas to contend not just for time, but also for the degrees of freedom provided by multiple antennas. We show that even when the medium is already occupied by some nodes, nodes with more antennas can transmit concurrently without harming the ongoing transmissions. Furthermore, such nodes can contend for the medium in a fully distributed way. Our testbed evaluation shows that even for a small network with three competing node pairs, the resulting system about doubles the average network throughput. It also maintains the random access nature of today's 802.11n networks.

The throughput of existing MIMO LANs is limited by the number of antennas on the AP. This paper shows how to overcome this limit. It presents interference alignment and cancellation (IAC), a new approach for decoding concurrent sender-receiver pairs in MIMO networks. IAC synthesizes two signal processing techniques, interference alignment and interference cancellation, showing that the combination applies to scenarios where neither interference alignment nor cancellation applies alone. We show analytically that IAC almost doubles the throughput of MIMO LANs. We also implement IAC in GNU-Radio, and experimentally demonstrate that for 2x2 MIMO LANs, IAC increases the average throughput by 1.5x on the downlink and 2x on the uplink.

This paper presents ZigZag, an 802.11 receiver design that combats hidden terminals. ZigZag's core contribution is a new form of interference cancellation that exploits asynchrony across successive collisions. Specifically, 802.11 retransmissions, in the case of hidden terminals, cause successive collisions. These collisions have different interference-free stretches at their start, which ZigZag exploits to bootstrap its decoding.

Traditionally, interference is considered harmful. Wireless networks strive to avoid scheduling multiple transmissions at the same time in order to prevent interference. This paper adopts the opposite approach; it encourages strategically picked senders to interfere. Instead of forwarding packets, routers forward the interfering signals. The destination leverages network-level information to cancel the interference and recover the signal destined to it. The result is analog network coding because it mixes signals not bits.

The performance of Transmission Control Protocol (TCP) over Ad hoc wireless networks (or simply ad hoc networks) has been extensively studied through simulations by the research community. Although many theoretical models, such as [1], have been proposed for estimating the performance of TCP over wired networks, researchers have faced many difficulties in modeling TCP over ad hoc networks. These difficulties are mainly due to the behavior of the underlying physical and MAC layers. Recently, [2] attempted to solve this problem by simplifying the behavior of TCP, besides assuming that no packet losses occur. In this work, we attempt to provide a theoretical model for TCP by considering the main phases of TCP, namely the slow start phase and the congestion avoidance phase, thus providing a more accurate model that captures all of its main features. To the best of our knowledge, ours is the first model that considers the slow start phase while analyzing TCP's performance in ad hoc networks. We make use of multi-dimensional Markovian chains to model each of these phases. We then use the resulting steady state probabilities to estimate the goodput. Furthermore, the analysis is validated by comparing the theoretical and simulation results using various error models.