Current touch-based interactions on earphones are limited by hygiene concerns and the small interaction surface. Recent works attempt to bypass these issues with mid-air gesture systems using active acoustic sensing. However, these signals may be audible and pose potential hearing risks. To address this, we propose FingerBar, a mid-air gesture recognition system for earphones that relies solely on microphones without active signal transmission. FingerBar leverages the distinctive friction sounds generated by finger gestures to achieve gesture recognition. We design a gesture filtering pipeline to maintain robustness against daily noise. An adversarial training strategy further enhances user-independent performance. From a set of 16 gestures, we identify the 7 most suitable for FingerBar based on user acceptability. Extensive evaluations demonstrate high accuracy and robustness. Furthermore, a user study confirms the practicality and acceptability of the system. Our findings highlight the promise of passive acoustic sensing as a user-friendly interaction modality for earphones.

Tear Film Break-Up Time (TBUT) is a critical clinical parameter in the management of dry eye disease (DED). However, traditional TBUT assessments rely on costly and time-consuming clinical procedures, while existing home-based solutions fail to provide precise TBUT values. In this work, we present Blinic, a contactless system leveraging commercial millimeter-wave (mmWave) radar to predict precise TBUT values and assess DED severity grades at home. Blinic incorporates detailed blink kinematics that are closely linked to TBUT. To address the challenge of predicting TBUT directly from radar data, we propose a teacher-student learning framework. The teacher model, trained on electronic health records (EHRs) including image-based diagnostic tests, transfers medical insights to the student model, which uses radar-captured blink dynamics. This knowledge transfer is further enhanced by a fine-tuned large language model, DryEye-LLM, which is based on clinical diagnostic reports and employs unsupervised domain adaptation to align EHRs with radar data. To ensure accurate blink motion capture, Blinic employs an antenna-coded MIMO mmWave radar design. Additionally, a query-based multitask learning module simultaneously predicts TBUT and DED severity grades, addressing potential conflicts in feature representation. Evaluated on 192 participants in collaboration with an eye clinic, Blinic demonstrates achieving a mean absolute error of 2.73 seconds for TBUT with an average accuracy of 90.54% for DED grading in real-world settings, providing a practical solution for home-based DED management.

As a crucial tool for assessing health, spirometry provides valuable insights into pulmonary functions. Recent advancements have enabled more convenient measurements by shifting spirometry solutions from cumbersome clinical devices to portable devices. However, the forced maneuvers and burdensome procedures, which necessitate repeated maximal forced breathing, often lead to dizziness and discomfort, rendering them unsuitable for vulnerable populations. In this paper, we present ESPIRO (Earphone-enabled Speech sPIROmetry) system to furnish user-friendly pulmonary function monitoring for diverse populations. Basically, ESPIRO records normal speech using microphone-embedded earphones and characterizes pulmonary function-related glottal flow during speech production. ESPIRO advances existing spirometry solutions in i) leveraging phonetics to associate pulmonary function with glottal flow in normal speech, thereby eliminating the need for forced breathing; ii) identifying effective speech features according to physiological basis, ensuring reliable spirometry measurements; and iii) effectively addressing ambient noise, making it suitable for various real-world settings. Extensive experiments with 38 subjects on 18 commodity earphones confirm that ESPIRO accurately estimates pulmonary function indices in practice.

Conventional pulmonary function tests (PFTs) are important but costly. Hence, prior research has proposed IoT sensor-based solutions to facilitate cost-efficient, at-home PFT. However, these solutions require the subject to perform maximal exhalations, a task often challenging without supervision, compromising test accuracy. In response to this challenge, this study introduces EasySpiro that, for the first time, uses non-maximal exhalations to measure PFT indicators. This is challenging since PFT indicators are only defined for maximal exhalations, and there are no guidelines to derive them from submaximal exhalations. To address that, we observe that pulmonary deficiencies affect all types of breathing, where the underlying pulmonary deficiency should be the same under different breathing efforts. Leveraging this insight, we design a reconstruction model to predict the ideal maximal breathing patterns based on submaximal ones and utilize these reconstructions for PFT. Furthermore, since the body dynamics reflect the exhalation effort, we use self-supervised learning techniques to encode body dynamics into breathing effort representations to guide the reconstruction process. We integrate these designs into earphones with microphones to measure breathing patterns and IMUs to measure body dynamics. We collaborate with a hospital and develop a dataset from 50 patients with various diseases to evaluate EasySpiro's performance, which shows an accurate prediction of PFT indicators based on non-maximal exhalations with an error rate of 7%. In addition, we open-source the collected dataset to encourage future research.

Respiratory diseases, highlighted by the COVID-19 pandemic, are now one of the leading causes of global mortality. Auscultation is an essential diagnostic tool for these conditions. However, its precision is dependent on the proficiency of healthcare providers, which can be a limitation in areas with scarce medical resources. Recently, the advent of sophisticated electronic stethoscopes and the application of deep learning have propelled the development of respiratory sound classification technologies to aid in clinical diagnosis. Yet, the variability of clinical auscultation environments has hindered the performance of these technologies. In this study, we propose a novel approach to enhance the model's performance through multi-task learning (MTL), simultaneously capturing the acoustic features of different respiratory sound types and their corresponding auscultation points. We've observed that normal respiratory sounds differ based on auscultation points, and the scarcity of data can confound models in discerning whether feature variations are point-related or indicative of pathology. By integrating both learning tasks, we overcome this challenge. To reinforce model robustness, we employ hyper-gradient descent (HD) to balance the task weights. Our work achieves a score of 62. 98% in the ICBHI data set, which is the arithmetic mean of Specificity and Sensitivity, surpassing the baseline by 3.43 % and demonstrating state-of-the-art performance. We believe that our findings can serve as inspiration and be integrated with other works employing data augmentation techniques to further enhance the performance and generalizability of models in clinical settings at large.

Motion tracking is an important aspect of human-computer interaction (HCI) and recent research focuses on motion tracking using earphones' embedded acoustic sensors. However, these solutions can only be deployed on wired ear-phones, while most of the commercial earphones are wireless ones. This limitation arises because wireless earphones utilize the Bluetooth Low Energy (BLE) protocol for handling audio data, which blocks the usage of existing acoustic sensing solutions. Firstly, the low sampling rate of BLE prevents the system from processing high-frequency ultrasounds. However, the sensing signal for earphones must be ultrasonic to prevent disturbance to the user. Secondly, BLE employs an audio compression process that is applied with different compression rates with different bandwidths. This will break the structure of wideband signals usually used for acoustic sensing. To overcome these challenges, we present BLEAR, the first earphone-tracking system compatible with the BLE audio recording protocol. To let BLE earphones receive ultrasounds, BLEAR utilizes a specially designed bandwidth conversion scheme that uses a mask signal to trigger a non-linear effect that converts high-frequency components to low-frequency ones, thereby overcoming the low audio sampling rate restriction of BLE. Additionally, by strategically designing beacon signals to align with BLE's subband compression pattern, BLEAR mitigates the influence of audio compression and achieves accurate wireless earphone tracking. We implement a wireless earphone prototype for BLEAR and conduct extensive experiments involving 8 subjects to demonstrate its feasibility. The experimental results show that BLEAR achieves a mean distance tracking error of 3.37 cm, an angle tracking error of 5.3 degrees, and an accuracy of 97.14% in recognizing 7 common user activities. This work not only introduces a BLE-compatible earphone tracking solution but also establishes a foundation for broader BLE device tracking applications.

In-time disease assessment is essential to better customize the medication scheme and improve the quality of life for chronic diseases like Parkinson's disease (PD). Toward the inconvenience problem in current clinical assessment practice, mobile sensing solutions based on detecting Parkinson's vocal changes are proposed. However, current solutions either can only achieve binary disease detection task or require patients to perform specific speaking tasks, which is not effective and practical for disease stage assessment in daily scenario. Moreover, most of existing solutions do not take speech privacy into consideration. In this work, we present PDAssess, a free speech-based daily assessment system that can perform 4-stage Parkinson's disease assessment in a privacy-preserving manner. We observe that current solutions did not fully leverage the rich information embedded in free speech due to the linguistic content variations, and therefore leverage a pre-trained automatic speech recognition (ASR) model to achieve a content variation-aware feature-extraction. In order to distinguish subtle stage-wise differences, we design a novel attention-based neural network architecture with a customized loss function for disease assessment task. Towards the potential privacy leakage problem, we design a Split Learning-based framework with pseudo-labeling and local domain adversarial training to better preserve speech content privacy. We collaborate with a medical center and evaluate the performance of PDAssess on real-world speech data collected from 50 PD subjects and 50 healthy subjects. The evaluation result shows that PDAssess can perform 4-stage PD assessment with an average person-wise F1 score of 89.1% and voice sample-wise F1 score of 75.1%.

The traditional interaction method for smart eyewear is by touching a control panel located at the temple front of the eyeglass. This method can be unnatural since the control panel and the display are not within the same plane. In this paper, we propose a new and natural interaction technology for smart eyewear that allows users to interact with the rim of the eyeglass without adding additional hardware to the rim. This design is based on an observation that a finger touch would slightly alter the channel frequency response (CFR) of the eyeglass. We use one pair of piezoelectric (PZT) transducers to measure the CFR, and we recognize the tiny CFR changes by analyzing the complex representation of the CFR. The system detects five touch locations using a deep learning classifier. We recruit ten subjects to evaluate the system and the result shows that the system can recognize the five touch locations with an F1 score of 0.91.

Recent works have shown that acoustic signals can be leveraged to perform respiration monitoring with high accuracy and low energy consumption. Since smartphones, smart speakers, and many other IoT devices are already equipped with microphones and speakers, it is convenient to implement the acoustic sensing solutions on those devices. However, the existing technologies require the speaker to transmit certain ultrasonic signals to detect respiration. Although these signals are inaudible to adults, they are audible to children and pets and they may even have negative impacts on plants. In this article, instead of using ultrasonic signals, we are trying to leverage audible signals in daily lives, e.g., music or broadcasting audios, to detect human respiration. We design a respiration detection system which derives the respiration rate by continuously estimates the channel impulse response (CIR) using music and broadcast signals. We study the intersymbol interference (ISI) brought by the randomness of music and broadcast signal and give our strategy to minimize the interference. We also propose several techniques to resolve some practical issues, such as the multipath effect and sampling frequency offset between the speaker and the microphone. Extensive experiments are conducted to demonstrate the feasibility of our system. The result shows that our system can achieve high respiration detection accuracy with the mean error of less than 0.5 BPM when different audio signals are used.