Camera-based respiratory monitoring is contactless, non-invasive, unobtrusive, and easily accessible compared to conventional wearable devices. This paper presents a novel multimodal approach to estimating breathing rate based on tracking the movement and color changes of the face through an RGB camera. A machine learning model determines the final breathing rate between two separately calculated ones from breathing motion and remote photoplethysmography (rPPG) to improve the measurement performance in a broader range of breathing frequencies. Our proposed pipeline is evaluated with 140 facial video recordings from 22 healthy subjects, including 6 controlled and 2 spontaneous breathing tasks ranging from 5 to 30 BPM. The estimation accuracy achieves 1.33 BPM mean absolute error and 86.53% pass rate within 2 BPM error criteria. To the best of our knowledge, our approach outperforms previous works that use a face region alone with a single RGB camera.

Stress can potentially have detrimental effects on both physical and mental well-being, but monitoring it can be challenging, especially in free-living conditions. One approach to address this challenge is to use earbud accelerometers to capture the ballistocardiogram (BCG) response. These sensors allow for noninvasive stress monitoring by estimating physiological indicators linked to stress, such as heart rate variability (HRV). However, ear-worn devices are susceptible to motion artifacts and can exhibit significant BCG signal morphology variations. These challenges necessitate accurate algorithms to estimate HRV for everyday use. Therefore, we developed a method to measure interbeat intervals (IBI) from BCG signals collected from an earbud. To enhance IBI estimation accuracy, we employed a Bayesian method that incorporates robust apriori IBI prediction weighting and sensor fusion techniques. We have also conducted a study involving 97 participants to assess the earbuds' ability to estimate HRV metrics and classify stressful activities. Our findings demonstrate low IBI estimation error (4.16% ± 1.90%), along with lower errors in subsequent higher-order HRV metrics compared to the state-of-the-art algorithms.

Core body temperature (CBT) is one of the critical yet under-explored phenomena in the context of stress detection. Several CBT measurement methods exist, but they are often limited in continuous CBT monitoring. Furthermore, how continuous CBT can be used to model acute stress is little explored. We address these challenges by conducting an in-lab controlled study with 97 participants who participated in baseline and stress-inducing tasks while wearing prototype earbuds capable of collecting CBT. We found that accounting for changes from individual baselines in CBT results is acute stress detection with 94.88% accuracy and 94.4% F1-score, which is 29.31% and 26.07% higher in terms of accuracy and F1-score, respectively, compared to generalized features.

Consumer hearable technologies such as earbuds are increasingly embedding physiological sensors, including photoplethysmography (PPG) and inertial measurements. They create unique opportunities to passively monitor stress and deliver digital interventions such as music. However, PPG signals recorded from ear canals are often very noisy due to head movement and fit issues. This work proposes algorithms to estimate heart rate variability (HRV) features from noisy PPG signals recorded using earbuds. We have used template matching to determine the signal quality for dynamic fine filtering around the estimated heart rate. We have also improved the inter-beat interval (IBI) outlier detection and removal algorithm using the global-local context of the input PPG signal. The mean absolute error of estimating RMSSD decreased from 70.83 milliseconds (ms) to 24.88 ms, and SDNN decreased from 46.89 ms to 16.60 ms.

The accurate estimation of peripheral capillary oxygen saturation (SpO2) is vital for monitoring respiratory health, with applications spanning medical diagnostics and fitness tracking. Remote photoplethysmography (rPPG) offers a convenient and non-contact approach for SpO2estimation. However, existing methods predominantly rely on data within the normal SpO2range, hindering their effectiveness during hypoxemia. Moreover, cross-user variations poses significant challenges for practicality. To address these limitations, we propose a simple yet effective normalization-based SpO2estimation algorithm. By aligning individual Ratio-of-Ratios (RoR) data with a standard model at the matching SpO2level, we mitigate cross-user variation, accommodate different camera configurations, and account for lighting changes. Our experiments demonstrate that the proposed method achieves an rMSE of 2.8% with leave-one-subject-out cross-validation across the full SpO2range (70%-100%), significantly outperforming existing RoR-based and CNN-based SpO2estimation approaches. Notably, our methods excel in accurately identifying hypoxemia, a critical clinical requirement. We anticipate broader applicability of our approach in rPPG-based vital sign monitoring, underlining the potential for enhancing robustness and reliability in various domains.

Camera-based pulse measurements from remote photoplethysmography (rPPG) have rapidly improved over recent years due to innovations in video processing and deep learning. However, modern data-driven solutions require large training datasets collected under diverse conditions. Collecting such training data is made more challenging by the need for time-synchronized video and physiological signals as ground truth. This paper presents a weakly supervised learning framework, Freq2Time, to train with heart rate (HR) labels. Our framework mitigates the need for simultaneous PPG or ECG as ground truth, since the HR changes relatively slowly and describes the target rPPG signal over a time interval. We show that 3D convolutional neural network (3DCNN) models trained with the Freq2Time framework give state-of-the-art HR performance with MAE of 2.86 bpm, when tested with challenging smartphone video data from 30 subjects. Additionally, our models still learn accurate rPPG time signals, allowing for other physiological metrics such as heart rate variability.

Tidal volume (VT) is defined as the volume of inhaled and exhaled air during normal breath, which is crucial for maintaining respiratory function, such as adequate air exchange in and out of the body. However, existing estimation methods either require complex setups or involve inconvenient and expensive devices, such as spirometer and chestband. Thus, leveraging the advanced artificial intelligence (AI) and wearable devices, we aim to develop a novel, accessible and convenient approach to estimate tidal volume. In this study, we propose the VTMonitor system, which utilizes consumer earbuds’ motion sensor data to estimate the tidal volume. We conducted two experiments, collecting data either in lab or at home. After analyzing the data, our VTMonitor system is effective in measuring the tidal volume.

The emergence of remote vital sign measurement techniques has provided an alternative approach for monitoring vital signs without direct physical contact. However, the performance of contactless methods such as remote photoplethysmography are dependent on the accuracy of face detection algorithms. The misalignment of the face pixels from frame to frame can introduce jitters in the generated rPPG signals, and in turn, interfere with vital sign estimations. Nonetheless, the investigation into the performance of face detectors mostly focused on quantifying the accuracy of face landmarks based on an individual image. How to assess facial alignments across video frames has largely remained unknown and understudied. To address this issue, this paper introduced three novel metrics for assessing face alignment in remote vital sign detection: (1) Circular Radius, (2) Mean Offset, and (3) Percentage of Impacted Pixels. We evaluated two face detectors using proposed metrics in static and motion scenarios, where static represents no facial movement and motion scenarios involve facial movements induced by breathing. Our experiments demonstrated that employing the superior face detector recommended by our metrics resulted in a noteworthy 12.5% reduction in second-level mean absolute error and a corresponding 3.0% improvement in 5%-accuracy for remote heart rate estimation.

Human Activity Recognition (HAR) is one important digital health applications to track fitness or to avoid sedentary behavior. Due to the growing popularity of consumer wearable devices, smartwatches and earbuds are being widely adopted for HAR applications. However, using just one of the devices may not be sufficient to track all activities properly. Additionally, handling motion noise becomes more challenging when a single device is used. This paper proposes a multi-modal approach to HAR by using both buds and watch. Using a large dataset of 53 subjects collected from both controlled and uncontrolled noisy environments, we demonstrate the limitations of using a single modality activity classification. We identify various noise sources imposed in uncontrolled environment and propose two novel noise handling methods to ensure the robustness of activity state tracking. We build on top of a previous activity tracking effort and demonstrate a 7.8% sensitivity improvement against current state of the art in uncontrolled noisy environment.

Breathing rate is critical for the user’s respiratory health and is hard to track outside the clinical context, requiring specialized devices. Earables could provide a convenient solution to track the breathing rate anywhere by leveraging the user’s breathing-related motion and sound captured through the earables’ motion sensors and microphones. However, small non-breathing head movements or background noises during the assessment affect the estimation accuracy. While noise filtering improves accuracy, it can discard valid measurements. This paper presents a multimodal approach to tracking the user’s breathing rate using a signal-processing-based algorithm on motion sensors and a lightweight machine-learning algorithm on acoustic sensors from the earables that balances the accuracy and data retention. A user study with 30 participants shows that the system can accurately calculate breathing rate (Mean Absolute Error < 2 breaths per minute) while retaining most breathing sessions (75%) performed in real-world settings. This work provides an essential direction for remote breathing rate monitoring.

Mouth breathing has been linked to a variety of negative health outcomes, including sleep-related disorders and dental problems. Detecting mouth breathing in the daily environment could be helpful for early intervention and reversing the negative impact. However, existing research has not adequately explored methods for detecting mouth breathing in everyday settings. This study presents a machine-learning approach using audio captured by commercially available earbuds to detect mouth breathing. By leveraging the growing popularity of earbuds for health monitoring, this approach offers a more convenient and non-invasive means of detecting mouth breathing. We conducted a data collection study with 30 participants to train a convolutional neural network-based model, which achieved an accuracy of 78.4% in detecting mouth breathing. Our findings suggest that audio-based mouth breathing detection using earbuds could be a promising tool for early intervention and improved health outcomes.

This paper presents algorithms to improve the estimation of heart rate (HR) and heart rate variability (HRV) from smartphone video. The remote photoplethysmogram (rPPG) signals are first extracted from the videos recorded. Next, we proposed an rPPG filter adaptively tuned by HR and respiratory rate (RR) to better enhance source signal that modulates HR. Additionally, we also addressed a unique smartphone artifact—occasionally seen in smartphone videos—by introducing a threshold-based algorithm. HR and HRV accuracies are assessed on 22 subjects who were instructed to breath at seven different RRs. The mean absolute errors of HR and standard deviation of the NN intervals (SDNN) are found to be 1.13 ± 0.68 bpm and 18.30 ± 10.33 ms respectively. Finally, we also conduct a few experiments to highlight the accuracy improvements made by the proposed algorithms.

Breathing Inhale/Exhale (IE) ratio is one of the critical breathing biomarkers for pulmonary patients and healthy individuals. It can indicate the severity of lung obstruction for chronic lung patients and help detect psycho-social stress for healthy individuals. With the advancement of wearable technologies, common consumer wearables such as smartwatches offer breathing rates. However, IE ratio measurement is not available in consumer wearable devices till today. In this paper, we present a novel algorithm, BreathIE, to estimate breathing rate and IE ratio using a low-power motion sensor embedded in consumer-grade earbuds. Moreover, our algorithm is adaptive and dynamically adjusts to the user’s breathing habit by accommodating varying breathing durations at run time. We conducted a study with 30 participants, where both earbuds and a reference chestband device were used simultaneously. Experimental evaluation against the annotated reference data shows that our algorithm can estimate breathing rate with a mean absolute error (MAE) of 2.37 breaths per minute (BPM) and breathing IE ratio of 0.27 MAE while outperforming the state-of-the-art algorithms.

Respiration rate is an important health biomarker and a vital indicator for health and fitness. With smart earbuds gaining popularity as a commodity device, recent works have demonstrated the potential for monitoring breathing rate using such earable devices. In this work, for the first time we utilize deep image recognition techniques to infer respiration rate from earbud audio. We use image spectrograms from breathing cycle audio signals captured using Samsung earbuds as a spectral feature to train a deep convolutional neural network. Using novel earbud audio data collected from 30 subjects with both controlled breathing at a wide range (from 5 upto 45 breaths per minute), and uncontrolled natural breathing from 7-day home deployment, experimental results demonstrate that our model outperforms existing methods using earbuds for inferring respiration rates from regular intensity breathing and heavy breathing sounds with 0.77 aggregated MAE for controlled breathing and with 0.99 aggregated MAE for at-home natural breathing.

With the popularity of non-invasive vital signs detection, remote photoplethysmography (rPPG) is drawing attention in the community. Remote PPG or rPPG signals are extracted in a contactless manner that is more prone to artifacts than PPG signals collected by wearable sensors. To develop a robust and accurate pipeline to estimate heart rate (HR) from rPPG signals, we propose a novel real-time dynamic ROI tracking algorithm that applies to slight motions and light changes. Furthermore, we develop and include a signal quality index (SQI) to improve the HR estimation accuracy. Studies have explored optimal SQIs for PPG signals, but not for remote PPG signals. In this paper, we select and test six SQIs: Perfusion, Kurtosis, Skewness, Zero-crossing, Entropy, and signal-to-noise ratio (SNR) on 124 rPPG sessions from 30 participants wearing masks. Based on the mean absolute error (MAE) of HR estimation, the optimal SQI is selected and validated by Mann–Whitney U test (MWU). Lastly, we show that the HR estimation accuracy is improved by 29% after removing outliers decided by the optimal SQI, and the best result achieves the MAE of 2.308 bpm.

Contactless measurement of vitals has been seen as a promising alternative to contact sensors for monitoring of health condition. In this paper, we focus on respiration rate (RR) as one of the fundamental biomarkers of a person’s cardio and pulmonary activities. Remote RR estimation has gained attraction due to its various potential applications; use of RGB cameras to extract remote photoplethysmography (PPG) signal from subjects’ face has been debated as one of the enabling technologies for remote RR estimation. The technology is challenged with respect to wide range of RR and non-voluntary motion in uncontrolled settings. We propose a novel methodology to enhance the remote PPG signal and remove artifacts from the respiration signal. The method achieves 3.9bpm MAE of 90% percentile (1.3bpm decrease) for estimating RR in range of 5-25bpm. We validate the performance using smartphone video recordings of 30 subjects with uniform distribution of skin tone.

Tracking breathing phases (inhale and exhale) outside the hospitals can offer significant health and wellness benefits. For example, the breathing phases can provide fine-grained breathing information for breathing exercises. While previous works use smartphones and smartwatches for tracking breathing phases, in this work, we use earbuds for breathing phase detection, which can be a better form factor for breathing exercises as it requires less user attention from the user. We propose a convolutional neural network-based algorithm for detecting breathing phases using the audio captured through the earbuds during guided breathing sessions. We conducted a user study with 30 participants in both lab and home environments to develop and evaluate our algorithm. Our algorithm can detect the breathing phases with 85% accuracy by taking only a 500ms audio signal. Our work demonstrates the potential of using earbuds for tracking the breathing phases in real-time.

We describe a novel computational framework for contactless oxygen saturation (SpO2) detection using videos recorded from human faces using smartphone cameras with ambient light. For contact pulse oximeter, a ratio of ratios (RoR) metric derived from selected regions of interest (ROI) combined with linear regression modeling is the standard approach. However, when used upon contactless remote PPG (rPPG), the assumptions of this standard approach do not hold automatically: 1) the rPPG signal is usually derived from the face area where the light reflection may not be uniform due to variation in skin tissue composition and/or lighting conditions (moles, hairs, beard, partial shadowing, etc.), 2) for most consumer-level cameras under ambient light, the rPPG signal is converted from light reflection associated with wide-band spectra, which creates complicated nonlinearity for SpO2mappings. We propose a computational framework to overcome these challenges by 1) determining and dynamically tracking the ROIs according to both spatial and color proximity, and calculating the RoR based on selected individual ROIs which have homogeneous skin reflections, and 2) using a nonlinear machine learning model to mapping the SpO2levels from RoRs derived from two different color combinations. We validated the framework with 30 healthy participants during various breathing tasks and achieved 1.24% Root Mean Square Error for across-subjects model and 1.06% for within-subject models, which surpassed the FDA-recognized ISO 81060-2-61:2017 standard.

Lung health assessment is traditionally done mainly through X-ray images and spirometry tests which are time-consuming, cumbersome, and costly. In this paper, we investigate the potential of passively recordable contents such as speech, cough and heart signal for such an assessment. Our regression model is the first in the literature to achieve mean absolute error (MAE) of 7.47% for estimation of forced expiratory volume in 1 sec. (FEV1) over forced vital capacity (FVC) ratio using these contents. This is comparable to the state of the art active phone-based spirometry methods. Additionally our classification models achieve a F1-score of 0.982 for healthy v.s. diseased, 0.881 for obstructive v.s. non-obstructive, 0.854 for chronic obstructive pulmonary disease (COPD) v.s. asthma, and 0.892 for severe v.s. non-severe obstruction classification.

Persistent coughs are a major symptom of respiratory-related diseases. Increasing research attention has been paid to detecting coughs using wearables, especially during the COVID-19 pandemic. Microphone is most widely used sensor to detect coughs. However, the intense power consumption needed to process audio hinders continuous audio-based cough detection on battery-limited commercial wearables, such as earbuds. We present CoughTrigger, which utilizes a lower-power sensor, inertial measurement unit (IMU), in earbuds as a cough detection activator to trigger a higher-power sensor for audio processing and classification. It runs all-the-time as a standby service with minimal battery consumption and triggers the audio-based cough detection when a candidate cough is detected from IMU. Besides, the use of IMU brings the benefit of improved specificity of cough detection. Experiments are conducted on 45 subjects and CoughTrigger achieved 0.77 AUC score. We also validated its effectiveness on free-living data and through on-device implementation.

Breathing exercises reduce stress and improve overall mental well-being. There are various types of breathing exercises. Performing the exercises correctly may give the best outcome and doing it in wrong ways can sometimes have adverse effect. Providing real-time biofeedback can greatly improve the user experience in doing the right exercises in the right ways. In this paper, we present methods to passively track breathing biomarkers in real-time using wireless commodity earbuds and generate feedback on users' breathing performance. We use the earbud's low-power accelerometer to generate a comprehensive set of breathing biomarkers including breathing phase, breathing rate, depth of breathing, and breathing symmetry. We have conducted studies where the subjects performed different types of guided breathing exercises while wearing the earbuds. Our algorithms detect breathing phases with 90.91% F1-score and estimate breathing rate with 95.05% accuracy. We further show that our algorithms can be used to generate biofeedback towards designing engaging smartphone's user interactions that facilitate users to accurately perform various breathing exercises.

Atrial Fibrillation (AF) is an important cardiac rhythm disorder, which if left untreated can lead to serious complications such as a stroke. AF can remain asymptomatic, and it can progressively worsen over time; it is thus a disorder that would benefit from detection and continuous monitoring with a wearable sensor. We develop an AF detection algorithm, deploy it on a smartwatch, and prospectively and comprehensively validate its performance on a real-world population that included patients diagnosed with AF. The algorithm showed a sensitivity of 87.8% and a specificity of 97.4% over every 5-minute segment of PPG evaluated. Furthermore, we introduce novel algorithm blocks and system designs to increase the time of coverage and monitor for AF even during periods of motion noise and other artifacts that would be encountered in daily-living scenarios. An average of 67.8% of the entire duration the patients wore the smartwatch produced a valid decision. Finally, we present the ability of our algorithm to function throughout the day and estimate the AF burden, a first-of-this-kind measure using a wearable sensor, showing 98% correlation with the ground truth and an average error of 6.2%.

There has been an extensive amount of study on cough detection using acoustic features captured from smartphones and smartwatches in the past decade. However, the specificity of the algorithms has always been a concern when exposed to the unseen field data containing cough-like sounds. In this paper, we propose a novel sensor fusion algorithm that employs a hybrid of classification and template matching algorithms to tackle the problem of unseen classes. The algorithm utilizes in-ear audio signal as well as head motion captured by the inertial measurement unit (IMU). A clinical study including 45 subjects from healthy and chronic cough cohorts was conducted that contained various tasks including cough and cough-like body sounds in various conditions such as quiet/noisy and stationary/non-stationary. Our hybrid model was evaluated for sensitivity and specificity in these conditions using leave one-subject out validation (LOSOV) and achieved an average sensitivity of 83% for stationary tasks and an specificity of 91.7% for cough-like sounds reducing the false positive rate by 55%. These results indicate the feasibility and superiority of fusion in earbuds platforms for detection of cough events.

Breathing rate is an important vital sign and an indicator of overall health and fitness. Traditionally breathing is monitored using specialized devices such as chestband or spirometers which are uncomfortable for daily use. Recent works show the feasibility of estimating breathing rate using earbuds' motion sensors. However, non-breathing head motion is one of the biggest challenges for breathing rate estimation using earbuds. In this paper, we propose algorithms to estimate breathing rate in presence of non-breathing head motion using inertial sensors embedded in commodity earbuds. Using the chestband as a reference device, we show that our algorithms can estimate breathing rate in resting positions with error rate 2.34 breaths per minute (BPM). Our algorithms can handle passive head motion and reduce the error by 27.78%. Furthermore, our algorithms can handle active head motion and help reduce the error by 45.70% when intentional non-breathing head motion is present in the data segment. It can be a big stride towards passive breathing monitoring in daily life using commodity earbuds.

Inertial measurement unit (IMU) sensors are widely used in motion tracking for various applications, e.g., virtual physical therapy and fitness training. Traditional IMU-based motion tracking systems use 9-axis IMU sensors that include an accelerometer, gyroscope, and magnetometer. The magnetometer is essential to correct the yaw drift in orientation estimation. However, its magnetic field measurement is often disturbed by the ferromagnetic materials in the environment and requires frequent calibration. Moreover, most IMU-based systems require multiple IMU sensors to track the body motion and are not convenient for use. In this paper, we propose a novel approach that uses a single 6-axis IMU sensor of a consumer smartwatch without any magnetometer to track the user's 3D arm motion in real time. We use a recurrent neural network (RNN) model to estimate the 3D positions of both the wrist and the elbow from the noisy IMU data. Compared with the state-of-the-art approaches that use either the 9-axis IMU sensor or the combination of a 6-axis IMU and an extra device, our proposed approach significantly improves the usability and potential for pervasiveness by not requiring a magnetometer or any extra device, while achieving comparable results.

Atrial Fibrillation (AF) is an important medical condition that can be passively detected and tracked using a smartwatch. Diagnosis and monitoring of AF can be more effective and reliable if the smartwatch senses continuously, but this can lead to significant battery consumption by the LED in the photoplethysmography (PPG) sensor. In this paper, we explore the feasibility of leveraging downsampling to achieve energy-efficient AF detection. We collect data from participants with paroxysmal AF in real ambulatory free-living environments using a commercial smartwatch and separately study the impact of uniform downsampling and compressed sensing on AF detection. Our results reveal that downsampling enables the AF detection system to consume about 77.4% less LED power than the original sampling strategy without a significant performance drop.

Tracking the type and frequency of cough events is critical for monitoring respiratory diseases. Coughs are one of the most common symptoms of respiratory and infectious diseases like COVID-19, and a cough monitoring system could have been vital in remote monitoring during a pandemic like COVID-19. While the existing solutions for cough monitoring use unimodal (e.g., audio) approaches for detecting coughs, a fusion of multimodal sensors (e.g., audio and accelerometer) from multiple devices (e.g., phone and watch) are likely to discover additional insights and can help to track the exacerbation of the respiratory conditions. However, such multimodal and multidevice fusion requires accurate time synchronization, which could be challenging for coughs as coughs are usually concise events (0.3-0.7 seconds). In this paper, we first demonstrate the time synchronization challenges of cough synchronization based on the cough data collected from two studies. Then we highlight the performance of a cross-correlation based time synchronization algorithm on the alignment of cough events. Our algorithm can synchronize 98.9% of cough events with an average synchronization error of 0.046s from two devices.

Mobil respiratory assessments using commodity smartphones and smartwatches are unmet needs for patient monitoring at home. In this paper, we show the feasibility of using multimodal sensors embedded in consumer mobile devices for non-invasive, low-effort respiratory assessment. We have conducted studies with 228 chronic respiratory patients and healthy subjects, and show that our model can estimate respiratory rate with mean absolute error (MAE) 0.72$\pm$0.62 breath per minute and differentiate respiratory patients from healthy subjects with 90% recall and 76% precision when the user breathes normally by holding the device on the chest or the abdomen for a minute. Holding the device on the chest or abdomen needs significantly lower effort compared to traditional spirometry which requires a specialized device and forceful vigorous breathing. This paper shows the feasibility of developing a low-effort respiratory assessment towards making it available anywhere, anytime through users' own mobile devices.

Obesity, a serious public health concern worldwide, increases the risk of many diseases, including hypertension, stroke, and type 2 diabetes. To tackle this problem, researchers collect diverse types of data, which includes biomedical, behavioral and activity, and utilize machine learning techniques to mine hidden patterns for obesity status improvement prediction. While existing machine learning methods such as Recurrent Neural Networks (RNNs) provide exceptional results, it is challenging to discover hidden patterns of the sequential data due to the irregular observation time instances. Meanwhile, the lack of understanding of why those learning models are effective also limits further improvements on their architectures. Thus, we develop a RNN based time-aware architecture to handle irregular observation times and identify relevant feature extractions from longitudinal patient records for obesity status improvement pre-diction. Evaluations of real-world data involving activity data collected from wearables and electronic health records demonstrate that our proposed method can capture the underlying structures in users' time sequences with irregularities, and achieve an accuracy of 77% in predicting the obesity status improvement.