Activities of daily living (ADL) identification with wearables has significant implications in healthy lifestyle management and offers an important sensor-based supervised learning research benchmark. Most ADL studies use single-modality (i.e., single sensor type like smartwatch only or earbuds only) data while multi-sensor data fusion studies using early-stage fusion of modalities from multi-sensors and multi-devices are emerging. To improve classification performance and model interpretability, leveraging early-stage fusion, late-stage fusion and individual modalities, we introduced novel modality-aware dynamic fusion (MAD-Fusion) models for multi-sensor data-fusion-based ADL identification. Based on early-stage fusion, we incorporated conformal prediction for uncertainty quantification, uncertainty late-stage fusion for cross-modality interpretability, and multi-modal strength-aware classification module. Trained on 36 independent subjects and tested on 4 independent subjects from Samsung ADL dataset with multi-sensor (accelerometers and gyroscopes) and multi-device (earbuds and smartwatch), MAD-Fusion not only achieved the state-of-the-art classification performance (accuracy: 0.9504, F1-score: 0.9142), but also enabled better interpretability of contributions and uncertainties from different modalities. The additional contribution of each building block is validated systematically. Furthermore, we validated MAD-Fusion’s superior performance on two public datasets in multi-sensor single-device settings (UCI-HAR and USC-HAD datasets). On all three datasets, MAD-Fusion manifested statistically significantly superiority comparing against baselines of single-modality, early-stage fusion and late-stage fusion (p< 0.001). To conclude, the novel MAD-Fusion models improve the classification performance, uncertainty quantification and interpretability for the ADL identification and can be applied to broader supervised learning areas requiring high performance, rigorous uncertainty quantification and model interpretability.

Earbuds are instrumental in health monitoring but the orientation can variate among users, which may significantly impact the health-monitoring system generalizability. To study the effect of earbuds orientation heterogeneity and align kinematics across earbuds orientations, we collected a dataset with various rotations relative to a baseline orientation. We developed the coordinate transformation by estimating Euler angles in transformation matrices with either grid search or Markov Chain Monte Carlo (MCMC) sampling. Taking ~ 17 seconds with a personal laptop, the MCMC method accurately estimated the coordinate transformation matrices to enable the transformed tri-axial linear acceleration to better match the baseline tri-axial linear acceleration with an average relative error of 1.899% (0.186 m/s2) and a maximum relative error of 2.774% averaged over all test orientations. Using the estimated transformation matrices and Samsung dataset of identification of activities of daily living (ADL), we validated the statistically significant impact of earbuds orientation heterogeneity on ADL identification (p < 0.001), which can cause 14.0% reduction in mean accuracy and 18.7% reduction in mean macro-average F1-score. To sum up, the MCMC method developed can be applied in earbuds kinematics alignment to address orientation heterogeneity and enable better earbuds-based health monitoring.

Human activity recognition has been an established active research area within the past few decades. While many researchers have tried to estimate some of the gait and running parameters, none was successful to provide a full suite of running dynamics parameters using commodity devices. Earbuds with their unique placement (in line with center of mass) provide an opportunity for activity recognition that never existed before with other commodity devices. Taking advantage of this opportunity, this work proposes a multi-modal approach to measure running dynamics using fusion of earbuds and smartwatch. Collecting a large dataset of 53 subjects, we developed various regression models to identify running parameters such as speed, cadence, stride length, vertical oscillation and ground contact time. These parameters were estimated in both jog and walk conditions and were evaluated in different device and context settings. Our MAPE ranges from 6.04% to 11.54% for various parameters.

Human Activity Recognition (HAR) is one important digital health applications to track fitness or to avoid sedentary behavior. Due to the growing popularity of consumer wearable devices, smartwatches and earbuds are being widely adopted for HAR applications. However, using just one of the devices may not be sufficient to track all activities properly. Additionally, handling motion noise becomes more challenging when a single device is used. This paper proposes a multi-modal approach to HAR by using both buds and watch. Using a large dataset of 53 subjects collected from both controlled and uncontrolled noisy environments, we demonstrate the limitations of using a single modality activity classification. We identify various noise sources imposed in uncontrolled environment and propose two novel noise handling methods to ensure the robustness of activity state tracking. We build on top of a previous activity tracking effort and demonstrate a 7.8% sensitivity improvement against current state of the art in uncontrolled noisy environment.

Breathing rate is critical for the user’s respiratory health and is hard to track outside the clinical context, requiring specialized devices. Earables could provide a convenient solution to track the breathing rate anywhere by leveraging the user’s breathing-related motion and sound captured through the earables’ motion sensors and microphones. However, small non-breathing head movements or background noises during the assessment affect the estimation accuracy. While noise filtering improves accuracy, it can discard valid measurements. This paper presents a multimodal approach to tracking the user’s breathing rate using a signal-processing-based algorithm on motion sensors and a lightweight machine-learning algorithm on acoustic sensors from the earables that balances the accuracy and data retention. A user study with 30 participants shows that the system can accurately calculate breathing rate (Mean Absolute Error < 2 breaths per minute) while retaining most breathing sessions (75%) performed in real-world settings. This work provides an essential direction for remote breathing rate monitoring.

Mouth breathing has been linked to a variety of negative health outcomes, including sleep-related disorders and dental problems. Detecting mouth breathing in the daily environment could be helpful for early intervention and reversing the negative impact. However, existing research has not adequately explored methods for detecting mouth breathing in everyday settings. This study presents a machine-learning approach using audio captured by commercially available earbuds to detect mouth breathing. By leveraging the growing popularity of earbuds for health monitoring, this approach offers a more convenient and non-invasive means of detecting mouth breathing. We conducted a data collection study with 30 participants to train a convolutional neural network-based model, which achieved an accuracy of 78.4% in detecting mouth breathing. Our findings suggest that audio-based mouth breathing detection using earbuds could be a promising tool for early intervention and improved health outcomes.

Tracking breathing phases (inhale and exhale) outside the hospitals can offer significant health and wellness benefits. For example, the breathing phases can provide fine-grained breathing information for breathing exercises. While previous works use smartphones and smartwatches for tracking breathing phases, in this work, we use earbuds for breathing phase detection, which can be a better form factor for breathing exercises as it requires less user attention from the user. We propose a convolutional neural network-based algorithm for detecting breathing phases using the audio captured through the earbuds during guided breathing sessions. We conducted a user study with 30 participants in both lab and home environments to develop and evaluate our algorithm. Our algorithm can detect the breathing phases with 85% accuracy by taking only a 500ms audio signal. Our work demonstrates the potential of using earbuds for tracking the breathing phases in real-time.

Lung health assessment is traditionally done mainly through X-ray images and spirometry tests which are time-consuming, cumbersome, and costly. In this paper, we investigate the potential of passively recordable contents such as speech, cough and heart signal for such an assessment. Our regression model is the first in the literature to achieve mean absolute error (MAE) of 7.47% for estimation of forced expiratory volume in 1 sec. (FEV1) over forced vital capacity (FVC) ratio using these contents. This is comparable to the state of the art active phone-based spirometry methods. Additionally our classification models achieve a F1-score of 0.982 for healthy v.s. diseased, 0.881 for obstructive v.s. non-obstructive, 0.854 for chronic obstructive pulmonary disease (COPD) v.s. asthma, and 0.892 for severe v.s. non-severe obstruction classification.

Persistent coughs are a major symptom of respiratory-related diseases. Increasing research attention has been paid to detecting coughs using wearables, especially during the COVID-19 pandemic. Microphone is most widely used sensor to detect coughs. However, the intense power consumption needed to process audio hinders continuous audio-based cough detection on battery-limited commercial wearables, such as earbuds. We present CoughTrigger, which utilizes a lower-power sensor, inertial measurement unit (IMU), in earbuds as a cough detection activator to trigger a higher-power sensor for audio processing and classification. It runs all-the-time as a standby service with minimal battery consumption and triggers the audio-based cough detection when a candidate cough is detected from IMU. Besides, the use of IMU brings the benefit of improved specificity of cough detection. Experiments are conducted on 45 subjects and CoughTrigger achieved 0.77 AUC score. We also validated its effectiveness on free-living data and through on-device implementation.

Breathing exercises reduce stress and improve overall mental well-being. There are various types of breathing exercises. Performing the exercises correctly may give the best outcome and doing it in wrong ways can sometimes have adverse effect. Providing real-time biofeedback can greatly improve the user experience in doing the right exercises in the right ways. In this paper, we present methods to passively track breathing biomarkers in real-time using wireless commodity earbuds and generate feedback on users' breathing performance. We use the earbud's low-power accelerometer to generate a comprehensive set of breathing biomarkers including breathing phase, breathing rate, depth of breathing, and breathing symmetry. We have conducted studies where the subjects performed different types of guided breathing exercises while wearing the earbuds. Our algorithms detect breathing phases with 90.91% F1-score and estimate breathing rate with 95.05% accuracy. We further show that our algorithms can be used to generate biofeedback towards designing engaging smartphone's user interactions that facilitate users to accurately perform various breathing exercises.

There has been an extensive amount of study on cough detection using acoustic features captured from smartphones and smartwatches in the past decade. However, the specificity of the algorithms has always been a concern when exposed to the unseen field data containing cough-like sounds. In this paper, we propose a novel sensor fusion algorithm that employs a hybrid of classification and template matching algorithms to tackle the problem of unseen classes. The algorithm utilizes in-ear audio signal as well as head motion captured by the inertial measurement unit (IMU). A clinical study including 45 subjects from healthy and chronic cough cohorts was conducted that contained various tasks including cough and cough-like body sounds in various conditions such as quiet/noisy and stationary/non-stationary. Our hybrid model was evaluated for sensitivity and specificity in these conditions using leave one-subject out validation (LOSOV) and achieved an average sensitivity of 83% for stationary tasks and an specificity of 91.7% for cough-like sounds reducing the false positive rate by 55%. These results indicate the feasibility and superiority of fusion in earbuds platforms for detection of cough events.

Breathing rate is an important vital sign and an indicator of overall health and fitness. Traditionally breathing is monitored using specialized devices such as chestband or spirometers which are uncomfortable for daily use. Recent works show the feasibility of estimating breathing rate using earbuds' motion sensors. However, non-breathing head motion is one of the biggest challenges for breathing rate estimation using earbuds. In this paper, we propose algorithms to estimate breathing rate in presence of non-breathing head motion using inertial sensors embedded in commodity earbuds. Using the chestband as a reference device, we show that our algorithms can estimate breathing rate in resting positions with error rate 2.34 breaths per minute (BPM). Our algorithms can handle passive head motion and reduce the error by 27.78%. Furthermore, our algorithms can handle active head motion and help reduce the error by 45.70% when intentional non-breathing head motion is present in the data segment. It can be a big stride towards passive breathing monitoring in daily life using commodity earbuds.

Obstructive pulmonary diseases cause limited airflow from the lung and severely affect patients' quality of life. Wheeze is one of the most prominent symptoms for them. High requirements imposed by traditional diagnosis methods make regular monitoring of pulmonary obstruction challenging, which hinders the opportunity of early intervention and prevention of significant exacerbation. In this work, we explore the feasibility of developing a mobile sensor-based system as a convenient means of assessing the severity of pulmonary obstruction via respiration phase-based symptomatic wheeze sensing. We conduct a 131 subjects' (91 patients and 40 healthy) study for the detection (F1: 87.96%) and characterization (F1: 79.47%) of wheeze. Subsequently, we develop novel wheeze metrics, which show a significant correlation (Pearson's correlation: -0.22, p-value: 0.024) with standard spirometry measure of pulmonary obstruction severity. This work takes a principal step towards the unobtrusive assessment of pulmonary condition from mobile sensor interactions.

Tracking the type and frequency of cough events is critical for monitoring respiratory diseases. Coughs are one of the most common symptoms of respiratory and infectious diseases like COVID-19, and a cough monitoring system could have been vital in remote monitoring during a pandemic like COVID-19. While the existing solutions for cough monitoring use unimodal (e.g., audio) approaches for detecting coughs, a fusion of multimodal sensors (e.g., audio and accelerometer) from multiple devices (e.g., phone and watch) are likely to discover additional insights and can help to track the exacerbation of the respiratory conditions. However, such multimodal and multidevice fusion requires accurate time synchronization, which could be challenging for coughs as coughs are usually concise events (0.3-0.7 seconds). In this paper, we first demonstrate the time synchronization challenges of cough synchronization based on the cough data collected from two studies. Then we highlight the performance of a cross-correlation based time synchronization algorithm on the alignment of cough events. Our algorithm can synchronize 98.9% of cough events with an average synchronization error of 0.046s from two devices.

Mobil respiratory assessments using commodity smartphones and smartwatches are unmet needs for patient monitoring at home. In this paper, we show the feasibility of using multimodal sensors embedded in consumer mobile devices for non-invasive, low-effort respiratory assessment. We have conducted studies with 228 chronic respiratory patients and healthy subjects, and show that our model can estimate respiratory rate with mean absolute error (MAE) 0.72$\pm$0.62 breath per minute and differentiate respiratory patients from healthy subjects with 90% recall and 76% precision when the user breathes normally by holding the device on the chest or the abdomen for a minute. Holding the device on the chest or abdomen needs significantly lower effort compared to traditional spirometry which requires a specialized device and forceful vigorous breathing. This paper shows the feasibility of developing a low-effort respiratory assessment towards making it available anywhere, anytime through users' own mobile devices.

Chronic respiratory diseases refer to a group of lung diseases that affect the airways and cause difficulty in breathing. Respiratory diseases are one of the leading causes of death and negatively impact the patients’ quality of life. Early detection and regular monitoring of lung functions might reduce the risk of death; however, lung function assessment requires the active supervision of a medical professional in a clinical setting. To make lung function tests more accessible and ubiquitous, researchers started leveraging mobile devices, which still require active supervision and demand extraneous effort from the user. In this work, we propose a convenient mobile-based approach that uses a monosyllabic voice segment called ‘A-vowel’ sound or ‘Aaaa...’ sound to estimate lung function. We conducted two studies (a lab study and an in-clinic study) with 201 participants to develop a detection model detecting ‘A-vowel’ sound from other acoustic events and a prediction model to estimate the lung function using the detected A-vowel sound. Our study shows that A-vowel sounds can be detected with 93% accuracy, and A-vowel sounds can estimate lung functions with 7.4-11.35% mean absolute error. We also conducted a validation study with 10 participants in a noisy environment and able to detect A-vowel segments with 71% F1-Score. Our results show auspicious directions to expand the horizon of mobile-based lung assessment.

Chronic obstructive pulmonary disease (COPD) and asthma are the most common respiratory diseases that impact millions of people worldwide annually. With advances in mobile computing and machine learning techniques, there has been increased interest in using mobile devices to monitor pulmonary diseases. Nevertheless, the current state-of-the-art technology requires active involvement and high-effort input from the users, impeding continuous monitoring of pulmonary conditions. In this work, two algorithms are proposed for passive assessment of pulmonary condition: one for detection of obstructive pulmonary disease and the other for estimation of the pulmonary function in terms of FEV1/FVC ratio, which is an established clinical metric. The algorithms were developed and validated using the data sets from two studies: research study (healthy=40, pathological=91) and in-clinic study (healthy=10, pathological=60). From the cross-study validation where a classifier was trained on the research data set and tested on the in-clinic data set, the detection accuracy of the pathological class was obtained as 73.7% and the F1 score was 84.5% (87.2% precision and 82.0% recall). In our regression analysis, the FEV1/FVC ratio was predicted with a mean absolute error of 8.6%. Our analysis shows promising results and this work presents a meaningful milestone towards the passive assessment of pulmonary functions from spontaneous speech collected from a mobile phone.

Spirometry is the gold standard to measure lung functions by estimating the maximum air an individual can forcefully exhale as quickly as possible. It is used not only to diagnose lung diseases such as asthma, chronic obstructive pulmonary disease (COPD) but also to assess the severity of the pulmonary condition. However, spirometry requires a specialized device called spirometer, which is mostly available in clinical facilities and cumbersome to use. Recent works have shown the feasibility of using smartphone microphone to estimate lung functions from forced exhalation effort sounds. However, maintaining the fidelity of lung function estimation on smartphones becomes challenging in unsupervised field environment in presence of other sounds such as coughs, deep inhalation, regular breathing, and speech. In this paper, we present ExhaleSense that detects forced exhalation efforts on smartphones from audio time-series data, distinguishes high fidelity efforts from poor efforts, and estimates lung obstruction. By conducting three studies with 211 pulmonary patients and healthy subjects, we show that ExhaleSense can detect forced exhalation sounds with 96.74% F1-score and estimate lung obstruction with mean absolute error as low as 7.57%. ExhaleSense shifts the gear of smartphone spirometry research from feasibility to ensuring effort quality towards high fidelity lung function estimation in unsupervised field settings.

Pulmonary disease is one of the leading causes of death, and individuals with chronic diseases like asthma and chronic obstructive pulmonary disease (COPD) will have to manage their disease throughout their lifetime. Passive monitoring with mobile devices such as smartphones represents a cost-effective solution that is closely coupled with the user to detect and continuously track adverse pulmonary trends. Identifying changes in the user's voice patterns has seen recent research interest as a potentially useful biomarker that is relatively convenient and efficient to monitor. Prosodic features of the voice such as shimmer and jitter have been traditionally extracted from sustained vowel sounds in the majority of the previous work in this area. In this work, we describe a method to extract these parameters from continuous running speech, and show that they have better agreement with the corresponding parameters from sustained vowel sounds. This method was validated on real data from pulmonary patients collected under two different environments, and also compared to an existing speech processing package that is widely used in the literature to extract such features. The described method can be an important step in realizing passive monitoring of voice changes from the natural speech of pulmonary patients in their day to day lives.

mLung is a privacy preserving, naturally windowed, mobile-cloud hybrid pulmonary care service for detecting unusual lung sounds like coughing and wheezing from streaming audio and inertial sensor data from a smartphone for pulmonary patients. mLung employs a combination of: (1) natural windowing of audio data from the patient respiration cycle captured by the inertial sensors, (2) in-phone speech detection and filtering by a lightweight classifier for patient privacy, and (3) in-cloud lung and confounding sound classification by a heavyweight and expert supervised classifier. This paper describes the design and architecture of mLung and using novel lung activity data collected by smartphone from 131 patients and healthy subjects, provides empirical evidence that mLung is 15%-25% more accurate in detecting lung sounds when compared to a state-of-the-art phone based internal body sound detection system using specialized microphone hardware, with a best f-1 score of 98%.

Chronic pulmonary disease is one of the leading causes of mortality in the United States. Continuous passive monitoring of subjects using mobile sensors can help detect disease, estimate severity, track progression over time, and predict adverse exacerbation events. One of the most convenient avenues to realize this goal is through analysis of passively recorded natural speech patterns. It has been previously established that diseases such as asthma and chronic obstructive pulmonary disease (COPD) affect pause patterns and prosodic features of speech. In this study we present an exploration of the feasibility of using speech features from natural speech to detect pulmonary disease. Experiments were conducted on a cohort of 131 subjects: 91 with asthma and/or COPD, and 40 healthy controls. Patients and healthy subjects were differentiable with 68% accuracy; moreover, the subset of patients with the highest disease severity were detected with 89% accuracy.

The prevalence of smart devices has enabled remote health monitoring outside of conventional clinical settings, and has reduced health care delivery cost. Passive audio recording is an essential component in remote health monitoring, however, it poses major privacy issues for subjects in uncontrolled environments like their home. There are existing voice activity detection and speech classification methodologies to identify sound events and obfuscate the human speech. However, they result in frequent false positives when distinguishing human speech from other sound events; their performance is limited to a controlled environment for a specific application; and require large amount of labeled data for training. In this paper, we present a novel speech privacy preservation methodology using generative adversarial networks to segment human speech in a recorded audio and generate human-like random speech to replace the original segment. We implemented our methodology and experimented on standard datasets of speech, environmental sounds, and cough samples generated from our internal mobile health study. Compared to current methodologies, our experimental results show much lower speech segmentation true positive rates of 17% and 14% for environmental sounds and cough datasets. Moreover, randomly generated audio samples to obfuscate the speech are shown to be likely indistinguishable from human speech (lower than 0.9% error in spectral attributes).

A gait velocity estimation algorithm using the inertial sensors of a smartwatch is proposed. The peaks of accelerometer and gyroscope norms are detected at first. Then a Kalman Filter is employed to recover the peaks that are missed because of the arm swing. The Kalman filter combines the accelerometer and gyroscope norm peaks and robustly detect walking step events even in cases where there is a large arm swing. Walking velocity is then estimated using the step duration. It will be shown in this work that the gait velocity has a good correlation with the inverse of the square of the step duration. The model parameters are calculated by collecting the training data from 25 subjects: each subject walked 50 m six times with different walking speed and different arm swing speed. The standard deviation of walking velocity estimation error is 0.1009 m/s (without person dependent calibration) and 0.0630 m/s (with person dependent calibration). The average precision of 91.7% was achieved for the gait speed testing on the smartwatch platform over all the speed scenarios.

This paper presents a methodology for developing highly-accurate, continuous Arterial Blood Pressure (ABP) models using only Photoplethysmography (PPG). In contrast to prior approaches, we develop a system that exhibits dynamic temporal behavior which leads to increased accuracy in modeling ABP. We validate our approach using data from patients in the intensive care unit (ICU). We show that it is possible to build highly accurate, continuous blood pressure models using only finger pulse oximeters. Our methodology achieves accurate systolic blood pressure estimation with a root mean square error 2.58 ± 1.23 across the patient sample used. Furthermore, the continuous ABP signal is estimated with a root mean square error of 6.042 ± 3.26 and correlation coefficient of 0.95 ± 0.045. Our method enables designing robust Remote Health Monitoring Systems (RMS) for Heart Failure patients without requiring traditional blood pressure monitors.

Poor adherence to prescription medication can compromise treatment effectiveness and cost the billions of dollars in unnecessary health care expenses. Though various interventions have been proposed for estimating adherence rates, few have been shown to be effective. Digital systems are capable of estimating adherence without extensive user involvement and can potentially provide higher accuracy with lower user burden than manual methods. In this paper, we propose a smartwatch-based system for detecting adherence to prescription medication based the identification of several motions using the built-in tri-axial accelerometers and gyroscopes. The efficacy of the proposed technique is confirmed through a survey of medication ingestion habits and experimental results on movement classification.