Tidal volume (VT) is defined as the volume of inhaled and exhaled air during normal breath, which is crucial for maintaining respiratory function, such as adequate air exchange in and out of the body. However, existing estimation methods either require complex setups or involve inconvenient and expensive devices, such as spirometer and chestband. Thus, leveraging the advanced artificial intelligence (AI) and wearable devices, we aim to develop a novel, accessible and convenient approach to estimate tidal volume. In this study, we propose the VTMonitor system, which utilizes consumer earbuds’ motion sensor data to estimate the tidal volume. We conducted two experiments, collecting data either in lab or at home. After analyzing the data, our VTMonitor system is effective in measuring the tidal volume.

Breathing rate is critical for the user’s respiratory health and is hard to track outside the clinical context, requiring specialized devices. Earables could provide a convenient solution to track the breathing rate anywhere by leveraging the user’s breathing-related motion and sound captured through the earables’ motion sensors and microphones. However, small non-breathing head movements or background noises during the assessment affect the estimation accuracy. While noise filtering improves accuracy, it can discard valid measurements. This paper presents a multimodal approach to tracking the user’s breathing rate using a signal-processing-based algorithm on motion sensors and a lightweight machine-learning algorithm on acoustic sensors from the earables that balances the accuracy and data retention. A user study with 30 participants shows that the system can accurately calculate breathing rate (Mean Absolute Error < 2 breaths per minute) while retaining most breathing sessions (75%) performed in real-world settings. This work provides an essential direction for remote breathing rate monitoring.

Mouth breathing has been linked to a variety of negative health outcomes, including sleep-related disorders and dental problems. Detecting mouth breathing in the daily environment could be helpful for early intervention and reversing the negative impact. However, existing research has not adequately explored methods for detecting mouth breathing in everyday settings. This study presents a machine-learning approach using audio captured by commercially available earbuds to detect mouth breathing. By leveraging the growing popularity of earbuds for health monitoring, this approach offers a more convenient and non-invasive means of detecting mouth breathing. We conducted a data collection study with 30 participants to train a convolutional neural network-based model, which achieved an accuracy of 78.4% in detecting mouth breathing. Our findings suggest that audio-based mouth breathing detection using earbuds could be a promising tool for early intervention and improved health outcomes.

Breathing Inhale/Exhale (IE) ratio is one of the critical breathing biomarkers for pulmonary patients and healthy individuals. It can indicate the severity of lung obstruction for chronic lung patients and help detect psycho-social stress for healthy individuals. With the advancement of wearable technologies, common consumer wearables such as smartwatches offer breathing rates. However, IE ratio measurement is not available in consumer wearable devices till today. In this paper, we present a novel algorithm, BreathIE, to estimate breathing rate and IE ratio using a low-power motion sensor embedded in consumer-grade earbuds. Moreover, our algorithm is adaptive and dynamically adjusts to the user’s breathing habit by accommodating varying breathing durations at run time. We conducted a study with 30 participants, where both earbuds and a reference chestband device were used simultaneously. Experimental evaluation against the annotated reference data shows that our algorithm can estimate breathing rate with a mean absolute error (MAE) of 2.37 breaths per minute (BPM) and breathing IE ratio of 0.27 MAE while outperforming the state-of-the-art algorithms.

Respiration rate is an important health biomarker and a vital indicator for health and fitness. With smart earbuds gaining popularity as a commodity device, recent works have demonstrated the potential for monitoring breathing rate using such earable devices. In this work, for the first time we utilize deep image recognition techniques to infer respiration rate from earbud audio. We use image spectrograms from breathing cycle audio signals captured using Samsung earbuds as a spectral feature to train a deep convolutional neural network. Using novel earbud audio data collected from 30 subjects with both controlled breathing at a wide range (from 5 upto 45 breaths per minute), and uncontrolled natural breathing from 7-day home deployment, experimental results demonstrate that our model outperforms existing methods using earbuds for inferring respiration rates from regular intensity breathing and heavy breathing sounds with 0.77 aggregated MAE for controlled breathing and with 0.99 aggregated MAE for at-home natural breathing.

Tracking breathing phases (inhale and exhale) outside the hospitals can offer significant health and wellness benefits. For example, the breathing phases can provide fine-grained breathing information for breathing exercises. While previous works use smartphones and smartwatches for tracking breathing phases, in this work, we use earbuds for breathing phase detection, which can be a better form factor for breathing exercises as it requires less user attention from the user. We propose a convolutional neural network-based algorithm for detecting breathing phases using the audio captured through the earbuds during guided breathing sessions. We conducted a user study with 30 participants in both lab and home environments to develop and evaluate our algorithm. Our algorithm can detect the breathing phases with 85% accuracy by taking only a 500ms audio signal. Our work demonstrates the potential of using earbuds for tracking the breathing phases in real-time.

Lung health assessment is traditionally done mainly through X-ray images and spirometry tests which are time-consuming, cumbersome, and costly. In this paper, we investigate the potential of passively recordable contents such as speech, cough and heart signal for such an assessment. Our regression model is the first in the literature to achieve mean absolute error (MAE) of 7.47% for estimation of forced expiratory volume in 1 sec. (FEV1) over forced vital capacity (FVC) ratio using these contents. This is comparable to the state of the art active phone-based spirometry methods. Additionally our classification models achieve a F1-score of 0.982 for healthy v.s. diseased, 0.881 for obstructive v.s. non-obstructive, 0.854 for chronic obstructive pulmonary disease (COPD) v.s. asthma, and 0.892 for severe v.s. non-severe obstruction classification.

Persistent coughs are a major symptom of respiratory-related diseases. Increasing research attention has been paid to detecting coughs using wearables, especially during the COVID-19 pandemic. Microphone is most widely used sensor to detect coughs. However, the intense power consumption needed to process audio hinders continuous audio-based cough detection on battery-limited commercial wearables, such as earbuds. We present CoughTrigger, which utilizes a lower-power sensor, inertial measurement unit (IMU), in earbuds as a cough detection activator to trigger a higher-power sensor for audio processing and classification. It runs all-the-time as a standby service with minimal battery consumption and triggers the audio-based cough detection when a candidate cough is detected from IMU. Besides, the use of IMU brings the benefit of improved specificity of cough detection. Experiments are conducted on 45 subjects and CoughTrigger achieved 0.77 AUC score. We also validated its effectiveness on free-living data and through on-device implementation.

Breathing exercises reduce stress and improve overall mental well-being. There are various types of breathing exercises. Performing the exercises correctly may give the best outcome and doing it in wrong ways can sometimes have adverse effect. Providing real-time biofeedback can greatly improve the user experience in doing the right exercises in the right ways. In this paper, we present methods to passively track breathing biomarkers in real-time using wireless commodity earbuds and generate feedback on users' breathing performance. We use the earbud's low-power accelerometer to generate a comprehensive set of breathing biomarkers including breathing phase, breathing rate, depth of breathing, and breathing symmetry. We have conducted studies where the subjects performed different types of guided breathing exercises while wearing the earbuds. Our algorithms detect breathing phases with 90.91% F1-score and estimate breathing rate with 95.05% accuracy. We further show that our algorithms can be used to generate biofeedback towards designing engaging smartphone's user interactions that facilitate users to accurately perform various breathing exercises.

There has been an extensive amount of study on cough detection using acoustic features captured from smartphones and smartwatches in the past decade. However, the specificity of the algorithms has always been a concern when exposed to the unseen field data containing cough-like sounds. In this paper, we propose a novel sensor fusion algorithm that employs a hybrid of classification and template matching algorithms to tackle the problem of unseen classes. The algorithm utilizes in-ear audio signal as well as head motion captured by the inertial measurement unit (IMU). A clinical study including 45 subjects from healthy and chronic cough cohorts was conducted that contained various tasks including cough and cough-like body sounds in various conditions such as quiet/noisy and stationary/non-stationary. Our hybrid model was evaluated for sensitivity and specificity in these conditions using leave one-subject out validation (LOSOV) and achieved an average sensitivity of 83% for stationary tasks and an specificity of 91.7% for cough-like sounds reducing the false positive rate by 55%. These results indicate the feasibility and superiority of fusion in earbuds platforms for detection of cough events.

Breathing rate is an important vital sign and an indicator of overall health and fitness. Traditionally breathing is monitored using specialized devices such as chestband or spirometers which are uncomfortable for daily use. Recent works show the feasibility of estimating breathing rate using earbuds' motion sensors. However, non-breathing head motion is one of the biggest challenges for breathing rate estimation using earbuds. In this paper, we propose algorithms to estimate breathing rate in presence of non-breathing head motion using inertial sensors embedded in commodity earbuds. Using the chestband as a reference device, we show that our algorithms can estimate breathing rate in resting positions with error rate 2.34 breaths per minute (BPM). Our algorithms can handle passive head motion and reduce the error by 27.78%. Furthermore, our algorithms can handle active head motion and help reduce the error by 45.70% when intentional non-breathing head motion is present in the data segment. It can be a big stride towards passive breathing monitoring in daily life using commodity earbuds.

In recent years, the use and importance of visual communication through photos have grown considerably. However, we have little understanding of the alignment between the intentions of the photo posters and the reactions of viewers. To address this gap, we replicated previous work that studied the alignment of poster and outsider judgments of text posts by extending it to photo posts. In our study of 573 users across four social media platforms, we found that outsiders generally judge photo posts more positively than anticipated by posters. Examining viewer engagement on social media revealed that photos depicting family and friends receive fewer reactions. We apply our insight to propose novel solutions that can help users create a more positive digital presence by aligning their photo posts with the expectations of their audiences.

Camera based assistive technologies such as smart glasses can provide people with visual impairments (PVIs) information about people in their vicinity. Although such ‘visually available’ information can enhance one’s social interactions, the privacy implications for bystanders from the perspective of PVIs remains underexplored. Motivated by prior findings of bystanders’ perspectives, we conducted two online surveys with visually impaired (N=128) and sighted (N=136) participants with two ‘field-of-view’ (FoV) experimental conditions related to whether information about bystanders was gathered from the front of the glasses or all directions. We found that PVIs considered it as ‘fair’ and equally useful to receive information from all directions. However, they reported being uncomfortable in receiving some visually apparent information (such as weight and gender) about bystanders as they felt it was ‘impolite’ or ‘improper’. Both PVIs and bystanders shared concerns about the fallibility of AI, where bystanders can be misrepresented by the devices. Our finding suggests that beyond issues of social stigma, both PVIs and bystanders have shared concerns that need to be considered to improve the social acceptability of camera based assistive technologies.

Obstructive pulmonary diseases cause limited airflow from the lung and severely affect patients' quality of life. Wheeze is one of the most prominent symptoms for them. High requirements imposed by traditional diagnosis methods make regular monitoring of pulmonary obstruction challenging, which hinders the opportunity of early intervention and prevention of significant exacerbation. In this work, we explore the feasibility of developing a mobile sensor-based system as a convenient means of assessing the severity of pulmonary obstruction via respiration phase-based symptomatic wheeze sensing. We conduct a 131 subjects' (91 patients and 40 healthy) study for the detection (F1: 87.96%) and characterization (F1: 79.47%) of wheeze. Subsequently, we develop novel wheeze metrics, which show a significant correlation (Pearson's correlation: -0.22, p-value: 0.024) with standard spirometry measure of pulmonary obstruction severity. This work takes a principal step towards the unobtrusive assessment of pulmonary condition from mobile sensor interactions.

Tracking the type and frequency of cough events is critical for monitoring respiratory diseases. Coughs are one of the most common symptoms of respiratory and infectious diseases like COVID-19, and a cough monitoring system could have been vital in remote monitoring during a pandemic like COVID-19. While the existing solutions for cough monitoring use unimodal (e.g., audio) approaches for detecting coughs, a fusion of multimodal sensors (e.g., audio and accelerometer) from multiple devices (e.g., phone and watch) are likely to discover additional insights and can help to track the exacerbation of the respiratory conditions. However, such multimodal and multidevice fusion requires accurate time synchronization, which could be challenging for coughs as coughs are usually concise events (0.3-0.7 seconds). In this paper, we first demonstrate the time synchronization challenges of cough synchronization based on the cough data collected from two studies. Then we highlight the performance of a cross-correlation based time synchronization algorithm on the alignment of cough events. Our algorithm can synchronize 98.9% of cough events with an average synchronization error of 0.046s from two devices.

Mobil respiratory assessments using commodity smartphones and smartwatches are unmet needs for patient monitoring at home. In this paper, we show the feasibility of using multimodal sensors embedded in consumer mobile devices for non-invasive, low-effort respiratory assessment. We have conducted studies with 228 chronic respiratory patients and healthy subjects, and show that our model can estimate respiratory rate with mean absolute error (MAE) 0.72$\pm$0.62 breath per minute and differentiate respiratory patients from healthy subjects with 90% recall and 76% precision when the user breathes normally by holding the device on the chest or the abdomen for a minute. Holding the device on the chest or abdomen needs significantly lower effort compared to traditional spirometry which requires a specialized device and forceful vigorous breathing. This paper shows the feasibility of developing a low-effort respiratory assessment towards making it available anywhere, anytime through users' own mobile devices.

Chronic respiratory diseases refer to a group of lung diseases that affect the airways and cause difficulty in breathing. Respiratory diseases are one of the leading causes of death and negatively impact the patients’ quality of life. Early detection and regular monitoring of lung functions might reduce the risk of death; however, lung function assessment requires the active supervision of a medical professional in a clinical setting. To make lung function tests more accessible and ubiquitous, researchers started leveraging mobile devices, which still require active supervision and demand extraneous effort from the user. In this work, we propose a convenient mobile-based approach that uses a monosyllabic voice segment called ‘A-vowel’ sound or ‘Aaaa...’ sound to estimate lung function. We conducted two studies (a lab study and an in-clinic study) with 201 participants to develop a detection model detecting ‘A-vowel’ sound from other acoustic events and a prediction model to estimate the lung function using the detected A-vowel sound. Our study shows that A-vowel sounds can be detected with 93% accuracy, and A-vowel sounds can estimate lung functions with 7.4-11.35% mean absolute error. We also conducted a validation study with 10 participants in a noisy environment and able to detect A-vowel segments with 71% F1-Score. Our results show auspicious directions to expand the horizon of mobile-based lung assessment.

Spirometry is the gold standard to measure lung functions by estimating the maximum air an individual can forcefully exhale as quickly as possible. It is used not only to diagnose lung diseases such as asthma, chronic obstructive pulmonary disease (COPD) but also to assess the severity of the pulmonary condition. However, spirometry requires a specialized device called spirometer, which is mostly available in clinical facilities and cumbersome to use. Recent works have shown the feasibility of using smartphone microphone to estimate lung functions from forced exhalation effort sounds. However, maintaining the fidelity of lung function estimation on smartphones becomes challenging in unsupervised field environment in presence of other sounds such as coughs, deep inhalation, regular breathing, and speech. In this paper, we present ExhaleSense that detects forced exhalation efforts on smartphones from audio time-series data, distinguishes high fidelity efforts from poor efforts, and estimates lung obstruction. By conducting three studies with 211 pulmonary patients and healthy subjects, we show that ExhaleSense can detect forced exhalation sounds with 96.74% F1-score and estimate lung obstruction with mean absolute error as low as 7.57%. ExhaleSense shifts the gear of smartphone spirometry research from feasibility to ensuring effort quality towards high fidelity lung function estimation in unsupervised field settings.

Abstract Millions of apps available to smartphone owners request various permissions to resources on the devices including sensitive data such as location and contact information. Disabling permissions for sensitive resources could improve privacy but can also impact the usability of apps in ways users may not be able to predict. We study an efficient approach that ascertains the impact of disabling permissions on the usability of apps through large-scale, crowdsourced user testing with the ultimate goal of making recommendations to users about which permissions can be disabled for improved privacy without sacrificing usability. We replicate and significantly extend previous analysis that showed the promise of a crowdsourcing approach where crowd workers test and report back on various configurations of an app. Through a large, between-subjects user experiment, our work provides insight into the impact of removing permissions within and across different apps (our participants tested three apps: Facebook Messenger (N=218), Instagram (N=227), and Twitter (N=110)). We study the impact of removing various permissions within and across apps, and we discover that it is possible to increase user privacy by disabling app permissions while also maintaining app usability.

Various technologies have been developed to help make the world more accessible to visually impaired people, and recent advances in low-cost wearable and mobile computing are likely to drive even moreadvances. However, the unique privacy and security needs of visually impaired people remain largely unaddressed. We conducted an exploratory user study with 14 visually impaired participants to understand the techniques they currently use for protecting privacy, their remaining privacy concerns,and how new technologies may be able to help. The interviews explored privacy not only in the physical world (e.g., bystanders overhearing private conversations) and the online world (e.g., determining if a URL is legitimate), but also in the interface between the two (e.g. bystanders `shoulder-surfing' data from screens). The study revealed serious concerns that are not adequately solved by current technology, and suggested new directions for improving the privacy of this significant fraction of the population.

Smartphone apps today request permission to access a multitude of sensitive resources, which users must accept completely during installation (e.g., on Android) or selectively configure after installation (e.g., on iOS, but also planned for Android). Everyday users, however, do not have the ability to make informed decisions about which permissions are essential for their usage. For enhanced privacy, we seek to leverage crowdsourcing to find minimal sets of permissions that will preserve the usability of the app for diverse users. We advocate an efficient 'lattice-based' crowd-management strategy to explore the space of permissions sets. We conducted a user study (N = 26) in which participants explored different permission sets for the popular Instagram app. This study validates our efficient crowd management strategy and shows that usability scores for diverse users can be predicted accurately, enabling suitable recommendations.