Speech disfluency research is pivotal to accommodating atypical speakers in mainstream conversational technology. However, the lack of publicly available labeled and unlabeled datasets is a significant bottleneck to such research. While many works use pseudo dysfluency data with proxy labels and formulate a self-supervised task, we see merit in using real-world data. In this work, we consolidate the corpora of publicly available speech disfluency datasets with and without labels and propose DisfluentSiam – an efficient siamese network-based small-scale pretraining pipeline using task-specific data from multiple domains with only 10M trainable parameters. We show that with DisfluentSiam, we achieve an average of 15% boost in performance across five types of dysfluency event detection compared to direct wav2vec 2.0 embeddings. In particular, with only 4-5 mins of labeled data for fine-tuning, the DisfluentSiam demonstrates the advantage of task-specific pretraining with up to 25% higher accuracy.

The effect of environmental change in WiFi signal is a major obstacle for generalized WiFi-based activity recognition. In this paper, we propose a novel method for adapting a WiFi-based activity classifier, that is trained on a large-scale dataset for an environment, to a new environment with only one labeled sample per activity. To this end, we propose a novel representation extraction algorithm using the supervision of visual data during the training phase. Our proposed feature extraction explicitly learns the corresponding relation between WiFi signal and movement of human body parts. For environment adaptation, we propose a framework that relies on only one labeled sample per activity class in comparison with current state-of-the-art solutions which are not suitable for few-shot adaptation. We collect data from four volunteers from five different environments and show that our proposed solution is able to achieve 28% higher accuracy than state-of-the-art solutions for environmental adaptation.

The accuracy of an audio classifier drops when it is trained and tested in different conditions aka domains, e.g., different devices, different environments, or their combinations. Previous works have proposed audio domain adaptation techniques for a special case where the training data are recorded with a single source microphone and the model is applied to test data recorded with a different but single target microphone (i.e., single source to single target domain adaptation). In this paper, we solve a more generic and practical problem where the goal is to adapt models that are trained on data from more than one acoustic (i.e., multi-source domain adaptation). Unlike previous works, the proposed method does not assume availability of recording metadata (i.e., domain labels) in the training data---which makes the adaptation problem harder. To solve this, we propose the first multi-task deep neural network architecture to cluster audio samples according to their domain in an unsupervised way. Using the inferred domain information, we perform domain adaptation to remove biases due to domain heterogeneity from the machine learning model. We conduct extensive experiments on an empirical dataset that we collect from five domains as well as on a public dataset. Our results show that the proposed technique has a mean accuracy of 87% for domain discovery in a five domain scenario and its model adaptation step improves acoustic event classification accuracy by up to 21% when compared to state-of-the-art algorithms on datasets containing samples from multiple source domains.

Human sensing, motion trajectory estimation, and identification are central to a wide range of applications in many domains such as retail stores, surveillance, public safety, public address, smart homes and cities, and access control. Existing solutions either require facial recognition or installation and maintenance of multiple units, or they lack long-term re-identification capability. In this paper, we propose a novel system - called EyeFi- that combines WiFi and camera on a standalone device to overcome these limitations. EyeFi integrates a WiFi chipset to an overhead camera and fuses motion trajectories obtained from both vision and RF modalities to identify individuals. In order to do that, EyeFi uses a student-teacher model to train a neural network to estimate the Angle of Arrival (AoA) of WiFi packets from the CSI values. Based on extensive evaluation using real-world data, we observe that EyeFi improves WiFi CSI based AoA estimation accuracy by more than 30% and offers 3,800 times computational speed over the state-of-the-art solution. In a real-world environment, EyeFi's accuracy of person identification averages 75% when the number of people varies from 2 to 10.

gestures, i.e., activities and gestures that have a semantically meaningful name in English language, as opposed to arbitrary free-form gestures. Given a list of activities (only their names in English text), along with zero or more training examples (WiFi CSI values) per activity, Wi-Fringe is able to detect all activities at runtime. We show for the first time that by utilizing the state-of-the-art semantic representation of English words, which is learned from datasets like the Wikipedia (e.g., Google's word-to-vector [1]) and verb attributes learned from how a word is defined (e.g, American Heritage Dictionary), we can enhance the capability of WiFi-based named gesture recognition systems that lack adequate training examples per class. We propose a novel cross-domain knowledge transfer algorithm between radio frequency (RF) and text to lessen the burden on developers and end-users from the tedious task of data collection for all possible activities. To evaluate Wi-Fringe, we collect data from four volunteers in a multi-person apartment and an office building for a total of 20 activities. We empirically quantify the trade-off between the accuracy and the number of unseen activities.

In this paper, we propose a fundamentally different approach to acoustic event classification that exploits knowledge from the textual domain to deal with a well-known pain point in audio event classification---i.e., the lack of adequate training examples. We show that by exploiting existing context-aware semantic representation of English words (e.g., Google's word-to-vector [33]) that is generated from a massive amount of English texts on the Internet, it is possible to classify acoustic events even when there are a few or no training examples for a wide variety of sounds. Our approach works in application scenarios where a system wants to learn a number of predefined categories of acoustic events, but it does not have enough training examples per category, and/or for some categories, it does not have any training examples at all. We solve this problem by combining a robust audio representation step, followed by a cross-modal projection of the audio representation onto textual representation. Our approach is different from techniques such as one-shot and data augmentation that do not consider the cross-domain knowledge transfers. We develop a generic mobile application for audio event detection where a user can input a list of desired sound types, along with training audio clips for some of those sound types, and the system is able to recognize all types of sounds (at varying level of accuracy depending on the number of classes that do not have any training examples), which is not achievable by any existing audio classifier that we are aware of. We evaluate the performance of the proposed system on an empirical dataset [41] as well as by deploying the application in two real-world scenarios. The accuracy of the classifier lies between 60%-90% for a 6--10 class problem when the number of classes that do not have any training examples is varied between 2--5.

With the prevalence of smartphones, pedestrians and joggers today often walk or run while listening to music. Since they are deprived of their auditory senses that would have provided important cues to dangers, they are at a much greater risk of being hit by cars or other vehicles. In this paper, we build a wearable system that uses multichannel audio sensors embedded in a headset to help detect and locate cars from their honks, engine, and tire noises, and warn pedestrians of imminent dangers of approaching cars. We demonstrate that using a segmented architecture consisting of headset-mounted audio sensors, a front-end hardware platform that performs signal processing and feature extraction, and machine learning-based classification on a smartphone, we are able to provide early danger detection in real time, from up to 60 m away, and alert the user with low latency and high accuracy. To further reduce power consumption of the battery-powered wearable headset, we implement a custom-designed integrated circuit that is able to compute delays between multiple channels of audio with nW power consumption. A regression-based method for sound source localization, angle via polygonal regression, is proposed and used in combination with the IC to improve the granularity and robustness of localization.

Low Power Wide Area Networks (LPWANs) are an excellent fit to city-scale IoT applications becuase of their long range and a battery life of several years, and a data rate of 25-50kbps, which is sufficient to carry IoT traffic. However, a practical limitation of a LPWAN-based real-time wireless network is the duty-cycle limit imposed on the sub-1GHz band by the FCC. In this paper, we overcome this challenge by proposing the first duty-cycle-aware wireless link scheduling algorithm for real-time LPWANs that considers the urgency of the packets as well as the availability of the wireless channels. The proposed algorithm is implemented in a five-node, wide-area outdoor test-bed in multiple realworld scenarios. Simulation results are provided to quantify its performance under different settings (e.g. larger networks, variety of workloads, and multiple baselines). In both realworld deployments and simulations, the proposed algorithm outperforms standard scheduling algorithms in terms of link schedulability, deadline misses, and buffer size.

This demo is an implementation of our motion-triggered camera system that captures, processes, stores, and transmits 3D visual information of a real-world environment using a low-cost camera-based sensor system that is constrained by its limited processing capability, storage, and battery life. This system can be used in applications such as capturing and sharing 3D content in the social media, training people in different professions, and post-facto analysis of an event. This system uses off-the-shelf hardware and standard computer vision algorithms. Its novelty lies in the ability to optimally control camera data acquisition and processing stages to guarantee the desired quality of captured information and battery life. The design of the controller is based on extensive measurements and modeling of the relationships between the linear and angular motion of a camera and the quality of generated 3D point clouds as well as the battery life of the system. To achieve this, we 1) devise a new metric to quantify the quality of generated 3D point clouds, 2) formulate an optimization problem to find an optimal trigger point for the camera system and prolongs its battery life while maximizing the quality of captured 3D environment and 3) make the model adaptive so that the system evolves and its performance improves over time.

The Glimpse.3D is a body-worn camera that captures, processes, stores, and transmits 3D visual information of a real-world environment using a low-cost camera-based sensor system that is constrained by its limited processing capability, storage, and battery life. The 3D content is viewed on a mobile device such as a smartphone or a virtual reality headset. This system can be used in applications such as capturing and sharing 3D content in the social media, training people in different professions, and post-facto analysis of an event. Glimpse.3D uses off-the-shelf hardware and standard computer vision algorithms. Its novelty lies in the ability to optimally control camera data acquisition and processing stages to guarantee the desired quality of captured information and battery life. The design of the controller is based on extensive measurements and modeling of the relationships between the linear and angular motion of a body-worn camera and the quality of generated 3D point clouds as well as the battery life of the system. To achieve this, we 1) devise a new metric to quantify the quality of generated 3D point clouds, 2) formulate an optimization problem to find an optimal trigger point for the camera system that prolongs its battery life while maximizing the quality of captured 3D environment, and 3) make the model adaptive so that the system evolves and its performance improves over time.

In this paper, we study the overhearing problem of continuous acoustic sensing devices such as Amazon Echo, Google Home, or such voice-enabled home hubs, and develop a system called SoundSifter that mitigates personal or contextual information leakage due to the presence of unwanted sound sources in the acoustic environment. Instead of proposing modifications to existing home hubs, we build an independent embedded system that connects to a home hub via its audio input. Considering the aesthetics of home hubs, we envision SoundSifter as a smart sleeve or a cover for these devices. SoundSifter has hardware and software to capture the audio, isolate signals from distinct sound sources, filter out signals that are from unwanted sources, and process the signals to enforce policies such as personalization before the signals enter into an untrusted system like Amazon Echo or Google Home. We conduct empirical and real-world experiments to demonstrate that SoundSifter runs in real-time, is noise resilient, and supports selective and personalized voice commands that commercial voice-enabled home hubs do not.

With the prevalence of smartphones, pedestrians and joggers today often walk or run while listening to music. Since they are deprived of their auditory senses that would have provided important cues to dangers, they are at a much greater risk of being hit by cars or other vehicles. In this demonstration we present SEUS, a wearable system aimed at Sense Enhancement for Urban Safety. SEUS uses a three-stage architecture, consisting of headset mounted audio sensors, an embedded front-end for signal processing and feature extraction, and machine learning based classification on a smartphone, to provide early danger detection for pedestrians in real-time.