Activities of daily living (ADL) identification with wearables has significant implications in healthy lifestyle management and offers an important sensor-based supervised learning research benchmark. Most ADL studies use single-modality (i.e., single sensor type like smartwatch only or earbuds only) data while multi-sensor data fusion studies using early-stage fusion of modalities from multi-sensors and multi-devices are emerging. To improve classification performance and model interpretability, leveraging early-stage fusion, late-stage fusion and individual modalities, we introduced novel modality-aware dynamic fusion (MAD-Fusion) models for multi-sensor data-fusion-based ADL identification. Based on early-stage fusion, we incorporated conformal prediction for uncertainty quantification, uncertainty late-stage fusion for cross-modality interpretability, and multi-modal strength-aware classification module. Trained on 36 independent subjects and tested on 4 independent subjects from Samsung ADL dataset with multi-sensor (accelerometers and gyroscopes) and multi-device (earbuds and smartwatch), MAD-Fusion not only achieved the state-of-the-art classification performance (accuracy: 0.9504, F1-score: 0.9142), but also enabled better interpretability of contributions and uncertainties from different modalities. The additional contribution of each building block is validated systematically. Furthermore, we validated MAD-Fusion’s superior performance on two public datasets in multi-sensor single-device settings (UCI-HAR and USC-HAD datasets). On all three datasets, MAD-Fusion manifested statistically significantly superiority comparing against baselines of single-modality, early-stage fusion and late-stage fusion (p< 0.001). To conclude, the novel MAD-Fusion models improve the classification performance, uncertainty quantification and interpretability for the ADL identification and can be applied to broader supervised learning areas requiring high performance, rigorous uncertainty quantification and model interpretability.

Earbuds are instrumental in health monitoring but the orientation can variate among users, which may significantly impact the health-monitoring system generalizability. To study the effect of earbuds orientation heterogeneity and align kinematics across earbuds orientations, we collected a dataset with various rotations relative to a baseline orientation. We developed the coordinate transformation by estimating Euler angles in transformation matrices with either grid search or Markov Chain Monte Carlo (MCMC) sampling. Taking ~ 17 seconds with a personal laptop, the MCMC method accurately estimated the coordinate transformation matrices to enable the transformed tri-axial linear acceleration to better match the baseline tri-axial linear acceleration with an average relative error of 1.899% (0.186 m/s2) and a maximum relative error of 2.774% averaged over all test orientations. Using the estimated transformation matrices and Samsung dataset of identification of activities of daily living (ADL), we validated the statistically significant impact of earbuds orientation heterogeneity on ADL identification (p < 0.001), which can cause 14.0% reduction in mean accuracy and 18.7% reduction in mean macro-average F1-score. To sum up, the MCMC method developed can be applied in earbuds kinematics alignment to address orientation heterogeneity and enable better earbuds-based health monitoring.

Human activity recognition has been an established active research area within the past few decades. While many researchers have tried to estimate some of the gait and running parameters, none was successful to provide a full suite of running dynamics parameters using commodity devices. Earbuds with their unique placement (in line with center of mass) provide an opportunity for activity recognition that never existed before with other commodity devices. Taking advantage of this opportunity, this work proposes a multi-modal approach to measure running dynamics using fusion of earbuds and smartwatch. Collecting a large dataset of 53 subjects, we developed various regression models to identify running parameters such as speed, cadence, stride length, vertical oscillation and ground contact time. These parameters were estimated in both jog and walk conditions and were evaluated in different device and context settings. Our MAPE ranges from 6.04% to 11.54% for various parameters.

Camera-based respiratory monitoring is contactless, non-invasive, unobtrusive, and easily accessible compared to conventional wearable devices. This paper presents a novel multimodal approach to estimating breathing rate based on tracking the movement and color changes of the face through an RGB camera. A machine learning model determines the final breathing rate between two separately calculated ones from breathing motion and remote photoplethysmography (rPPG) to improve the measurement performance in a broader range of breathing frequencies. Our proposed pipeline is evaluated with 140 facial video recordings from 22 healthy subjects, including 6 controlled and 2 spontaneous breathing tasks ranging from 5 to 30 BPM. The estimation accuracy achieves 1.33 BPM mean absolute error and 86.53% pass rate within 2 BPM error criteria. To the best of our knowledge, our approach outperforms previous works that use a face region alone with a single RGB camera.

The accurate estimation of peripheral capillary oxygen saturation (SpO2) is vital for monitoring respiratory health, with applications spanning medical diagnostics and fitness tracking. Remote photoplethysmography (rPPG) offers a convenient and non-contact approach for SpO2estimation. However, existing methods predominantly rely on data within the normal SpO2range, hindering their effectiveness during hypoxemia. Moreover, cross-user variations poses significant challenges for practicality. To address these limitations, we propose a simple yet effective normalization-based SpO2estimation algorithm. By aligning individual Ratio-of-Ratios (RoR) data with a standard model at the matching SpO2level, we mitigate cross-user variation, accommodate different camera configurations, and account for lighting changes. Our experiments demonstrate that the proposed method achieves an rMSE of 2.8% with leave-one-subject-out cross-validation across the full SpO2range (70%-100%), significantly outperforming existing RoR-based and CNN-based SpO2estimation approaches. Notably, our methods excel in accurately identifying hypoxemia, a critical clinical requirement. We anticipate broader applicability of our approach in rPPG-based vital sign monitoring, underlining the potential for enhancing robustness and reliability in various domains.

Human Activity Recognition (HAR) is one important digital health applications to track fitness or to avoid sedentary behavior. Due to the growing popularity of consumer wearable devices, smartwatches and earbuds are being widely adopted for HAR applications. However, using just one of the devices may not be sufficient to track all activities properly. Additionally, handling motion noise becomes more challenging when a single device is used. This paper proposes a multi-modal approach to HAR by using both buds and watch. Using a large dataset of 53 subjects collected from both controlled and uncontrolled noisy environments, we demonstrate the limitations of using a single modality activity classification. We identify various noise sources imposed in uncontrolled environment and propose two novel noise handling methods to ensure the robustness of activity state tracking. We build on top of a previous activity tracking effort and demonstrate a 7.8% sensitivity improvement against current state of the art in uncontrolled noisy environment.

Breathing rate is critical for the user’s respiratory health and is hard to track outside the clinical context, requiring specialized devices. Earables could provide a convenient solution to track the breathing rate anywhere by leveraging the user’s breathing-related motion and sound captured through the earables’ motion sensors and microphones. However, small non-breathing head movements or background noises during the assessment affect the estimation accuracy. While noise filtering improves accuracy, it can discard valid measurements. This paper presents a multimodal approach to tracking the user’s breathing rate using a signal-processing-based algorithm on motion sensors and a lightweight machine-learning algorithm on acoustic sensors from the earables that balances the accuracy and data retention. A user study with 30 participants shows that the system can accurately calculate breathing rate (Mean Absolute Error < 2 breaths per minute) while retaining most breathing sessions (75%) performed in real-world settings. This work provides an essential direction for remote breathing rate monitoring.

Respiration rate is an important health biomarker and a vital indicator for health and fitness. With smart earbuds gaining popularity as a commodity device, recent works have demonstrated the potential for monitoring breathing rate using such earable devices. In this work, for the first time we utilize deep image recognition techniques to infer respiration rate from earbud audio. We use image spectrograms from breathing cycle audio signals captured using Samsung earbuds as a spectral feature to train a deep convolutional neural network. Using novel earbud audio data collected from 30 subjects with both controlled breathing at a wide range (from 5 upto 45 breaths per minute), and uncontrolled natural breathing from 7-day home deployment, experimental results demonstrate that our model outperforms existing methods using earbuds for inferring respiration rates from regular intensity breathing and heavy breathing sounds with 0.77 aggregated MAE for controlled breathing and with 0.99 aggregated MAE for at-home natural breathing.

Tracking breathing phases (inhale and exhale) outside the hospitals can offer significant health and wellness benefits. For example, the breathing phases can provide fine-grained breathing information for breathing exercises. While previous works use smartphones and smartwatches for tracking breathing phases, in this work, we use earbuds for breathing phase detection, which can be a better form factor for breathing exercises as it requires less user attention from the user. We propose a convolutional neural network-based algorithm for detecting breathing phases using the audio captured through the earbuds during guided breathing sessions. We conducted a user study with 30 participants in both lab and home environments to develop and evaluate our algorithm. Our algorithm can detect the breathing phases with 85% accuracy by taking only a 500ms audio signal. Our work demonstrates the potential of using earbuds for tracking the breathing phases in real-time.

Breathing exercises reduce stress and improve overall mental well-being. There are various types of breathing exercises. Performing the exercises correctly may give the best outcome and doing it in wrong ways can sometimes have adverse effect. Providing real-time biofeedback can greatly improve the user experience in doing the right exercises in the right ways. In this paper, we present methods to passively track breathing biomarkers in real-time using wireless commodity earbuds and generate feedback on users' breathing performance. We use the earbud's low-power accelerometer to generate a comprehensive set of breathing biomarkers including breathing phase, breathing rate, depth of breathing, and breathing symmetry. We have conducted studies where the subjects performed different types of guided breathing exercises while wearing the earbuds. Our algorithms detect breathing phases with 90.91% F1-score and estimate breathing rate with 95.05% accuracy. We further show that our algorithms can be used to generate biofeedback towards designing engaging smartphone's user interactions that facilitate users to accurately perform various breathing exercises.

Breathing rate is an important vital sign and an indicator of overall health and fitness. Traditionally breathing is monitored using specialized devices such as chestband or spirometers which are uncomfortable for daily use. Recent works show the feasibility of estimating breathing rate using earbuds' motion sensors. However, non-breathing head motion is one of the biggest challenges for breathing rate estimation using earbuds. In this paper, we propose algorithms to estimate breathing rate in presence of non-breathing head motion using inertial sensors embedded in commodity earbuds. Using the chestband as a reference device, we show that our algorithms can estimate breathing rate in resting positions with error rate 2.34 breaths per minute (BPM). Our algorithms can handle passive head motion and reduce the error by 27.78%. Furthermore, our algorithms can handle active head motion and help reduce the error by 45.70% when intentional non-breathing head motion is present in the data segment. It can be a big stride towards passive breathing monitoring in daily life using commodity earbuds.

Tracking the type and frequency of cough events is critical for monitoring respiratory diseases. Coughs are one of the most common symptoms of respiratory and infectious diseases like COVID-19, and a cough monitoring system could have been vital in remote monitoring during a pandemic like COVID-19. While the existing solutions for cough monitoring use unimodal (e.g., audio) approaches for detecting coughs, a fusion of multimodal sensors (e.g., audio and accelerometer) from multiple devices (e.g., phone and watch) are likely to discover additional insights and can help to track the exacerbation of the respiratory conditions. However, such multimodal and multidevice fusion requires accurate time synchronization, which could be challenging for coughs as coughs are usually concise events (0.3-0.7 seconds). In this paper, we first demonstrate the time synchronization challenges of cough synchronization based on the cough data collected from two studies. Then we highlight the performance of a cross-correlation based time synchronization algorithm on the alignment of cough events. Our algorithm can synchronize 98.9% of cough events with an average synchronization error of 0.046s from two devices.

Mobil respiratory assessments using commodity smartphones and smartwatches are unmet needs for patient monitoring at home. In this paper, we show the feasibility of using multimodal sensors embedded in consumer mobile devices for non-invasive, low-effort respiratory assessment. We have conducted studies with 228 chronic respiratory patients and healthy subjects, and show that our model can estimate respiratory rate with mean absolute error (MAE) 0.72$\pm$0.62 breath per minute and differentiate respiratory patients from healthy subjects with 90% recall and 76% precision when the user breathes normally by holding the device on the chest or the abdomen for a minute. Holding the device on the chest or abdomen needs significantly lower effort compared to traditional spirometry which requires a specialized device and forceful vigorous breathing. This paper shows the feasibility of developing a low-effort respiratory assessment towards making it available anywhere, anytime through users' own mobile devices.

Cities are deploying tens of thousands of sensors and actuators and developing a large array of smart services. The smart services use sophisticated models and decision-making policies supported by Cyber Physical Systems and Internet of Things technologies. The increasing number of sensors collects a large amount of city data across multiple domains. The collected data have great potential value, but has not yet been fully exploited. This survey focuses on the domains of transportation, environment, emergency and public safety, energy, and social sensing. This article carefully reviews both the data sets being collected across 14 smart cities and the state-of-the-art work in modeling and decision making methodologies. The article also points out the characteristics, challenges faced today, and those challenges that will be exacerbated in the future. Key data issues addressed include heterogeneity, interdisciplinary, integrity, completeness, real-timeliness, and interdependencies. Key decision making issues include safety and service conflicts, security, uncertainty, humans in the loop, and privacy.

mLung is a privacy preserving, naturally windowed, mobile-cloud hybrid pulmonary care service for detecting unusual lung sounds like coughing and wheezing from streaming audio and inertial sensor data from a smartphone for pulmonary patients. mLung employs a combination of: (1) natural windowing of audio data from the patient respiration cycle captured by the inertial sensors, (2) in-phone speech detection and filtering by a lightweight classifier for patient privacy, and (3) in-cloud lung and confounding sound classification by a heavyweight and expert supervised classifier. This paper describes the design and architecture of mLung and using novel lung activity data collected by smartphone from 131 patients and healthy subjects, provides empirical evidence that mLung is 15%-25% more accurate in detecting lung sounds when compared to a state-of-the-art phone based internal body sound detection system using specialized microphone hardware, with a best f-1 score of 98%.

We develop solutions to various challenges in different stages of the processing pipeline of a real time indoor distant speech emotion recognition system to reduce the discrepancy between training and test conditions for distant emotion recognition. We use a novel combination of distorted feature elimination, classifier optimization, several signal cleaning techniques and train classifiers with synthetic reverberation obtained from a room impulse response generator to improve performance in a variety of rooms with various source-to-microphone distances. Our comprehensive evaluation is based on a popular emotional corpus from the literature, two new customized datasets and a dataset made of YouTube videos. The two new datasets are the first ever distance aware emotional corpuses and we created them by 1) injecting room impulse responses collected in a variety of rooms with various source-to-microphone distances into a public emotional corpus; and by 2) re-recording the emotional corpus with microphones placed at different distances. The overall performance results show as much as 15.51% improvement in distant emotion detection over baselines, with a final emotion recognition accuracy ranging between 79.44%-95.89% for different rooms, acoustic configurations and source-to-microphone distances. We experimentally evaluate the CPU time of various system components and demonstrate the real time capability of our system.

The populations of large cities around the world are growing rapidly. Cities are beginning to address this problem by implementing significant sensing and actuation infrastructure and building services on this infrastructure. However, as the density of sensing and actuation increases and as the complexities of services grow there is an increasing potential for conflicts across Smart City services. These conflicts can cause unsafe situations and disrupt the benefits that the services were originally intended to provide. Although some of the conflicts can be detected and avoided during designing the services, many can still occur unpredictably during runtime. This paper carefully defines and enumerates the main issues regarding the detection and resolution of runtime conflicts in smart cities. In particular, it focuses on conflicts that arise across services. This issue is becoming more and more important as Smart City designs attempt to integrate services from different domains (transportation, energy, public safety, emergency, medical, and many others). Research challenges are identified and then addressed that deal with uncertainty, dynamism, real-time, mobility and spatio-temporal availability, duration and scale of effect, efficiency, and ownership. A watchdog architecture is also described that oversees the services operating in a Smart City. This watchdog solution detects and resolves conflicts, it learns and adapts, and it provides additional inputs to decision making aspects of services. Using data from a Smart City dataset, an emulated set of services and activities using those services are created to perform a conflict analysis. A second analysis hypothesizes 41 future services across 5 domains. Both of these evaluations demonstrate the high probability of conflicts in smart cities of the future.

Wireless Network-on-Chip (NoC) architectures have emerged as an enabling solution to design scalable NoC fabrics for massive many-core chips. However, such massive levels of integration of Intellectual Property (IP) cores make the chips vulnerable to malicious intrusions from untrustworthy processes or vendors. Hence, resilience to various types of hardware security threats is imperative in future many-core chips. In this paper we develop a design methodology to increase the resilience of a wireless NoC to Denial-of-Service (DoS) attacks. We demonstrate that the proposed architecture can sustain higher data transfer rates at lower energy dissipation with the spread of DoS attacks compared to conventional mesh based NoCs.