Camera-based blood pressure (BP) monitoring holds significant promise for non-invasive, continuous health assessment in clinical and daily-care settings. Pulse transit time (PTT), a key physiological indicator correlated with BP, has been widely explored for camera-BP estimation. We propose a new concept that measures the PTT based on the property of skin-tissue depolarization, termed skin-tissue depolarized PTT (SD-PTT). This method extracts PTT from camera-based photoplethysmography (PPG) signals at varying tissue depths based on a single skin site with a single wavelength. By employing time-division multiplexing to sequentially illuminate the skin with 0° and 90° polarized light, SD-PTT captures PPG signals from shallow tissues (via parallel polarization) and deeper tissues (via cross polarization), quantifying the phase delay between these signals, caused by different depolarization properties of skin layers as PTT for BP estimation. Laboratory experiments with cold and heat stimulation protocols (n = 20) reveal significant variations in SD-PTT that are consistent with the regulatory mechanisms of BP (P< 0.01). The SD-PTT is further validated in an intensive care unit scenario (n = 20) by modeling the relationship between SD-PTT and BP using a random forest regressor. Although the model performance is relatively limited under strict Leave-One-Out Cross-Validation (LOOCV), under a mixed modeling approach trained with 50% of personalized data, the method achieves remarkable SD-PTT-BP calibration performance, with mean absolute errors (MAE) of 4.85mmHg for systolic BP, 2.14mmHg for diastolic BP, and 2.85mmHg for mean BP, respectively. These results, achieved through 50% personalized calibration, meet the standards of the American Association for the Advancement of Medical Instrumentation (AAMI). The SD-PTT method offers a novel solution for camera-based monitoring, with strong potential for clinical applications.

The diagnosis of peripheral artery disease (PAD) typically relies on specialized equipment such as ultrasound. The delayed PAD detection of these approaches may lead to amputation and even death. To achieve rapid and ubiquitous PAD screening, we propose a novel concept of camera-based plantar perfusion imaging (CPPI) for PAD diagnosis and severity classification. Specifically, we performed a simulation trial that used an RGB camera to record the plantar video of 20 subjects and a cuff with different pressures applied to the left leg to simulate different degrees of lower limb blockage. We generated the plantar perfusion maps using remote photoplethysmography imaging and proposed a multi-view perfusion (MVP) feature set to represent the perfusion maps for PAD classification. The experimental results show that the Pearson correlation coefficients between MVP and Doppler ultrasound (clinical reference) features were larger than 0.9. MVP feature combined with Support Vector Machine obtains 91.47% accuracy in distinguishing the normal and obstructed states, and 76.48% accuracy in differentiating four different degrees of vascular obstruction. The clinical benchmark demonstrated the potential of CPPI as a rapid, sensitive, and easy-to-use diagnostic tool for PAD, suitable for large-scale screening in home or community settings.

Existing respiratory monitoring techniques primarily focus on respiratory rate measurement, neglecting the potential of using thoracoabdominal patterns of respiration for infant lung health assessment. To bridge this gap, we exploit the unique advantage of spatial redundancy of a camera sensor to analyze the infant thoracoabdominal respiratory motion. Specifically, we propose a camera-based respiratory imaging (CRI) system that utilizes optical flow to construct a spatio-temporal respiratory imager for comparing the infant chest and abdominal respiratory motion, and employs deep learning algorithms to identify infant abdominal, thoracoabdominal synchronous, and thoracoabdominal asynchronous patterns of respiration. To alleviate the challenges posed by limited clinical training data and subject variability, we introduce a novel multiple-expert contrastive learning (MECL) strategy to CRI. It enriches training samples by reversing and pairing different-class data, and promotes the representation consistency of same-class data through multi-expert collaborative optimization. Clinical validation involving 44 infants shows that MECL achieves 70% in sensitivity and 80.21% in specificity, which validates the feasibility of CRI for respiratory pattern recognition. This work investigates a novel video-based approach for assessing the infant thoracoabdominal patterns of respiration, revealing a new value stream of video health monitoring in neonatal care.

Continuous monitoring of cardiac activity is crucial for detecting anomalies such as heart failure and coronary artery disease, and it can alleviate the burden of cardiovascular disease on healthcare systems. This study introduces a novel concept for contactless monitoring of multi-point cardiac motion using dual-wavelength defocused speckle imaging (DW-DSI). A prototype system was developed to measure multi-point seismocardiography (MP-SCG) signals from the atrial and ventricular regions. In addition, blood pressure (BP) monitoring was demonstrated as a proof of concept using the time delay between atrial and ventricular motion signals. An experiment involving 19 subjects with ice water stimulation protocol demonstrated that the performance of BP estimation using time delay features of MP-SCG is comparable to BP estimated from ECG-PPG derived pulse arrival time. The results showed that the best performance was achieved using the correlation features extracted from MP-SCG, such as time delay information and heart rate, in combination with an artificial neural network model. The mean absolute error for systolic/diastolic/mean BP are 6.954 mmHg, 5.368 mmHg and 5.415 mmHg, with Pearson correlation coefficient of 0.639, 0.559, and 0.517. This demonstrates the potential of the camera-based DW-DSI system for measuring MP-SCG and the feasibility towards continuous BP monitoring.

Compared to broader physical activities, detecting nuanced respiratory movements poses a significant challenge in indoor health monitoring systems. While respiratory activity can be conceptualized as periodic chest movements akin to mechanical vibrations, uncontrollable environmental factors often introduce noise into detected radar signals. The clutter in the field of view, especially metallic objects, such as hospital steel beds, degrades the performance of radar physiological monitoring: 1) amplifying noise of multipath effects and 2) misleading the informative localization module. In this article, we propose a preprocessing scheme of Search-Voxel Ellipse Normalization for respiratory detection system, including an ellipse normalization method combined with the fitting-cost voxel selection policy, to improve the respiration detection performance using MIMO frequency modulated continuous wave radar. This article provides an in-depth assessment of the designed system, including a metal-insulated room test, involving ten participants in different postures. The results show notable performance improvements of our proposed ENDTW-MVMD method, especially in lowering the mean absolute error from the best state-of-the-art 0.93–0.75 bpm and stabilization in voxel selection. The proposed approach is thoroughly evaluated against established methods across various dimensions, such as voxel selection, independent performance, frequency estimation, and ablation studies.

In the Intensive Care Unit (ICU), monitoring a patient’s heart rate variability (HRV) can provide vital information regarding the physiological status and autonomic nervous system, serving as a potential metric for detecting adverse events in ICU. Although camera-based remote photoplethysmography (rPPG) has shown to be effective for heart rate monitoring in clinical environments, it remains insufficient research on utilizing rPPG-derived pulse rate variability (PRV) for monitoring autonomic nervous system (ANS) changes of ICU patients. In addition, the agreement between HRV and PRV is controversial, especially for ICU patients with multiple concurrent diseases. This study aims to validate the accuracy of camera-PRV measurements by analyzing eight HRV/PRV parameters from subjects with different health conditions, further discussing its feasibility as an early-warning tool. The accuracy of rPPG-derived PRV was comparable to that of contact-PPG and has high correlations with Electrocardiogram. Combined with machine learning algorithms, the system was able to accurately differentiate between healthy subjects and ICU patients (accuracy = 0.86, F1 score = 0.83, sensitivity = 0.79, specificity = 0.79). Furthermore, the system demonstrated good reliability in classifying subjects with different health status and ages. However, it could not effectively correspond to the APACHE II score when classifying ICU patients with different severities based on HRV/PRV. Because the APACHE II scoring system evaluates the changes in the patient’s ANS as well as a variety of confounding factors, of which the HRV/PRV parameters represent only a portion. In summary, the camera-PRV system was able to provide the individual physiological tracking and coarse-level modeling of the group health status. It is expected to achieve a more comprehensive health status assessment by integrating more vital signs in the future.

Blood pressure (BP) is a vital physiological parameter for early warning and prompt intervention treatment in the neonatal intensive care unit (NICU). However, the application of contactless BP measurement methods in neonates remains under-explored. This proof-of-concept clinical study proposes using multi-site and multi-wavelength pulse transit time (PTT) generated from remote-Photoplethysmography (rPPG) for neonatal BP estimation. A dataset of 40 neonates was created in the NICU under three alternating phases (resting -BP measurement -resting). The spatially averaged rPPG signals from five body parts were used to calculate multiple PTT features, including multi-site PTT (MS-PTT) derived from different body parts and multi-wavelength PTT (MW-PTT) derived from different skin layers, for BP estimation. Three machine learning models, including Multivariate Linear Regression (MLR), Support Vector Regression (SVR), and Random Forest Regression (RFR), were employed for both univariate and multivariate regression. Combining MS-PTT and MW-PTT yielded the best results, achieving a mean absolute error ± standard deviation (MAE ± STD) of 7.65 ± 7.48 mmHg for SBP, 6.31 ± 5.58 mmHg for DBP, and 7.29 ± 7.29 mmHg for MBP, based on MLR with subject-dependent modeling. According to the British Hypertension Society guidelines, these results meet the requirements for Grade C. These findings provide the first clinical proof-of-concept of using camera-based MS-PTT and MW-PTT features for contactless neonatal BP estimation.

The monitoring of respiratory rate (RR), heart rate (HR), HR variability (HRV), and blood pressure (BP) during sleep allows for a comprehensive evaluation of sleep quality, facilitating the understanding and improvement of a person’s sleep health. Contactless physiological monitoring using cameras has gained popularity recently due to its convenient, infection-free, continuous, and versatile nature. However, the privacy concerns limit the application of camera-based solutions in sleep monitoring setups. This study proposes a novel hybrid setup that integrates camera-based seismocardiography (CamSCG) and photoplethysmography (CamPPG) for contactless measurement of RR, HR, and HRV during sleep while simultaneously estimating BP. For the proximal SCG, we employed camera-based laser speckle vibrometry to measure cardiac motions from the chest, and benchmarked it with a millimeter-wave radar (RFSCG). For the distal photoplethysmographic (PPG), a defocused camera was utilized to measure pulse signals from the facial skin while protecting privacy. In this setup, we analyzed the single-modality in measuring RR, HR, and HRV, and established two bi-modalities (CamSCG-CamPPG and RFSCG-CamPPG) to measure pulse transit time (PTT) features for BP calibration. The benchmark involving 19 subjects highlights the potential of camera-based bi-modal SCG-PPG for privacy-protected vital signs monitoring during sleep.

Current camera-based infant monitoring mainly focuses on physiological measurement, overlooking its important semantic analysis potential for detecting accidental suffocation caused by oronasal occlusion during sleep. However, developing a robust infant suffocation risk detection model typically requires substantial labeled data, which is very difficult to obtain in real-world scenarios. To address this, we utilized the text-to-image diffusion model to generate diverse infant images depicting oronasal occlusion and non-occlusion scenarios controlled by text prompts. To ease the process of labeling, self- and semi-supervised learning algorithms are leveraged to learn the semantic information from unlabeled data with the support of minimal labeled data to train different model architectures. To evaluate the feasibility of this solution, we conducted a clinical trial in the neonatology department, which collected video data from 22 infants under various oronasal occlusion scenarios using breathable covers (e.g. clinical tissue). The clinical evaluation shows that most models trained on 25,000 generated images achieved over 90% performance on metrics of accuracy, recall, and F1-score, outperforming conventional approaches that pre-train and fine-tune the model using over 90,000 labeled task-related online images. This demonstrates the feasibility of leveraging text-to-image generated data to achieve robust camera-based infant suffocation risk detection, so as to secure the sleep safety of infants. More importantly, it beacons the potential of using text-based large-scale model to solve the general issue of scarcity of human data in artificial intelligence-based healthcare or clinical applications.

Respiratory spatial patterns describe the distribution and dynamics of lung conditions, and monitoring of their asymmetry or irregularities enables a more comprehensive assessment of respiratory functions. The feasibility of using the camera pixel array sensing combined with machine learning for analyzing respiratory spatial patterns was demonstrated, however, this approach faces challenges in patient generalization due to limited clinical data and individual respiratory variability. Data augmentation methods may address this by synthesizing new data, but they have a risk of destroying the symmetry or regular semantic information of respiratory patterns. To address this, we propose a prototype-driven hard-sample contrastive learning (PHCL) method tailored for camera-based respiratory imaging analysis. It first separates the samples into simple and hard-to-learn samples using prototypes and Gini-index distance measurement. Then it synthesizes a new feature by blending one-class simple samples and other-class hard samples to construct a transition boundary between different classes, so as to broaden the feature distribution. Then it employs contrastive learning to emphasize feature consistency between prototypes and hard same-class samples from different subjects to mitigate individual respiratory variability and refine class boundaries. Extensive experiments were conducted in the neonatal intensive care unit and the thoracic surgery department, where PHCL outperforms image augmentation and advanced feature augmentation methods by 1-10% in both accuracy and F1-score. Our work provides valuable insights into the analysis of asymmetric and irregular respiratory activities.

Conventional physiological monitoring relies on multiple contact-based biomedical sensors, such as electrocardiograms, pulse oximeters, and blood pressure (BP) cuffs, which often necessitate the use of cumbersome sensors (e.g., electrodes, patches, diodes) and extensive wiring. These conventional methods not only impose significant inconvenience on both caregivers and patients but also elevate the risk of patient infections. In this study, we introduce a novel non-contact, multi-parameter vital signs monitoring platform based on the red, green and near Infrared (RG-IR) spectral imaging system, which measures five critical patient parameters in the Intensive Care Unit (ICU), including heart rate (HR), heart rate variability (HRV), respiratory rate (RR), blood oxygen saturation (SpO$_{2}$) and BP. Clinical trials were conducted in the ICUs of two hospitals, involving 30 critically ill patients. The clinical outcomes indicate that our system achieves reasonable performance, with mean absolute errors of 1.89 bpm for HR, 19.04 ms for SDNN (standard deviation of normal-to-normal intervals), 0.99 bpm for RR, 2.75% for SpO$_{2}$ and 5.67-8.98 mmHg for BP, providing timely physiological information and marking a significant step toward contactless ICU monitoring. Additionally, the real-time performance of the core algorithms were validated on low-cost embedded chips, demonstrating the system's capability for edge computation and on-board patient monitoring. The proposed system has entered the registration process of the National Medical Products Administration (NMPA) as a Class II medical device (No. 2024007).

Cardiovascular diseases are one of the leading causes of death worldwide. Accurately capturing and analyzing the multidimensional dynamics of cardiac motion is crucial for early diagnosis and rehabilitation assessment. This study introduces a novel concept for non-contact cardiac linear vibration (SCG) and rotational components (GCGx and GCGy) decoupling and reconstruction by integrating speckle motion signals captured from two cameras with different defocus levels. The intention is to overcome the motion coupling issues inherent in single-camera imaging and improve the accuracy in characterizing the cardiac complex 3D mechanical behavior. Using a sternum-mounted inertial sensor as the reference, experiments were conducted on 42 subjects in laboratory and intensive care unit settings. The results show that the reconstructed cardiac 3D motion signals exhibit greater waveform similarity to the reference signal than the raw speckle motion signal from a single camera, with similarity indices above 87.471%. In addition, with an 8 ms tolerance error, the localization accuracy of 6 key biomarkers (aortic valve opening/closing (AO/AC), mitral valve opening/closing (MO/MC), the biomarkers corresponding to the AO event in GCGy and the MC event in GCGx) are 73.080%, 99.998%, 85.587%, 86.617%, 99.683% and 77.301%, respectively. These results also outperform those obtained from the raw speckle motion signal. These findings validate the rationale and effectiveness of using dual-camera imaging with different defocus levels to reconstruct SCG, GCGx, and GCGy, offering a promising approach for accurately capturing complex cardiac 3D motion and improving cardiac function assessment.

Cameras hold great potential in sleep monitoring and are suitable for long-term home use. However, privacy invasion is a major concern of cameras in sleep monitoring. A defocused camera offers a promising privacy-protection solution, its feasibility for physiological signal measurement has been shown but its performance on sleep staging remains unknown. We applied four levels of post-blurring (from sharp to blurred videos) to investigate the sleep staging including wakefulness (Wake), rapid eye movement (REM), light stage (Light), and deep stage (Deep). The results showed that all four levels of blurring effectively provide identity protection, but the sleep staging performance (especially the Light) was degraded by video blurring, particularly due to the distorted motion cues and high-frequency components of heart rate variability (HRV). To this end, we introduced Cardiopulmonary Coupling (CPC) to video-based sleep staging. With CPC, the accuracy (ACC) reached 69.5% (kappa = 0.59, F1-score = 0.67) with privacy protection, while under the most severe blurring, ACC was improved from 59.8% (kappa = 0.49, F1-score = 0.56) to 66.9% (kappa = 0.54, F1-score = 0.65). The main benefit of CPC is that the photoplethysmography (PPG) derived respiration (PDR) signal is more robust than inter-beat interval (IBI) under video blurring. Overall, this study explores the opportunity of using blurred videos for privacy-protected sleep staging and demonstrates the effectiveness of CPC in mitigating the degradation of motion and HRV features caused by video blurring. Cohort diversity and multimodal integration are expected to offer deeper clinical and technical insights for real-world applications.

Camera-based respiration monitoring is currently focused on the continuous measurement of respiratory rate, overlooking its potential in lung health assessment. Inspired by auscultation and palpation that use respiratory symmetry to assess the lung rehabilitation of thoracic surgery patients, we exploit the advantage of spatial redundancy of a camera sensor to replicate this clinical routine. In particular, we propose a camera-based respiratory imaging (CRI) system that leverages optical flow and deep learning algorithms to analyze the symmetric/asymmetric patterns of chest respiratory motion, and classify the subject as health, left lesion, or right lesion. To mitigate the issues of sample scarcity and subject variance, we introduce a novel multiple-prototype contrastive model (MPCM) that uses the symmetric respiration hypothesis to generate more training data, and produces multiple deep prototypes to enhance the consistency of deep representation of samples from different subjects. The clinical validation involving 45 subjects demonstrates the feasibility of CRI for lung rehabilitation assessment, where MPCM achieves above 70% in the used evaluation indices (e.g. accuracy, sensitivity, and specificity). This study demonstrates a new value stream in video health monitoring that uses camera-based respiratory imaging for the lung rehabilitation assessment of thoracic patients after surgery.

Heart rate variability (HRV) can be used as an effective indicator for the maturity of the neonatal autonomic nervous system (ANS) and some acute neonatal diseases (e.g., respiratory distress syndrome and late-onset sepsis). Advances in contactless sensing technology have led to the rapid development of camera-based physiological monitoring, but few studies have focused on HRV measurement in the Neonatal Intensive Care Unit. This study aims to evaluate the performance of camera-based neonatal HRV monitoring (camera-HRV) using camera–photoplethysmography (PPG), and to further investigate the feasibility of applying camera-HRV for tracking ANS maturation through quantitative analysis of 12 HRV indices on quiet infants with limited motions. The clinical experimental results show that as the ANS of newborns matures, the trends of camera-HRV indices are consistent with those obtained by electrocardiogram (ECG) and PPG. Furthermore, the performance of camera-HRV in predicting the postmenstrual age of newborns is comparable to that obtained by ECG and PPG, which can predict 63.89% of infants with an accuracy of less than ±2 weeks. In particular, camera-HRV shows excellent performance in the classification of preterm/term infants, with an optimal accuracy of 0.90 and an F1 score of 0.88. The pilot clinical study suggests that camera-HRV can be a feasible tool for tracking the development of neonatal ANS.

The continuous vital signs monitoring allows clinicians to timely assess the physiological conditions of the patient in intensive care unit (ICU). Internet of Things (IoT) in the hospital that integrated various sensors (e.g., cameras) may enable intelligent healthcare. In this article, we proposed a remote patient monitoring system that exploits closed-circuit television (CCTV) cameras in an IoT infrastructure as optical sensors for noncontact physiological measurement, extending its scope from surveillance to warding. The proposed monitoring system implemented the latest camera photoplethysmography algorithms for cardio-respiratory measurement, particularly focused on heart rate (HR) and breathing rate (BR) as fundamental biomarkers for earlier warning of deterioration. A clinical trial involving 25 critically ill patients was carried out in ICU, where the camera focal length (i.e., key impact factor) is thoroughly investigated. The clinical results show that our system achieves a mean absolute error (MAE) of 1.3 bpm for HR and 0.7 brpm for BR in the far-focus mode; an MAE of 1.0 bpm for HR and 1.8 brpm for BR in the near-focus mode, which are in the range of clinical acceptance. The comparison between the two modes suggests that the far-focus is more suitable for monitoring patient’s vital signs in this scenario. The success rate of HR and BR in the far-focus mode is 94.5% (MAE = 2 bpm) and 96.7% (MAE = 3 brpm), respectively. The prototypes show that the increased measurement coverage and convenience of CCTV cameras in an IoT system are useful for ubiquitous patient monitoring in hospital care units.

Most of the existing photoplethysmography (PPG)-based blood pressure (BP) estimation methods aim at training a general BP model applicable to all individuals which neglected the vasculature and anatomical differences among individuals as well as the slow and subtle cardiovascular changes over time, thus, were hard to achieve high accuracy. This study aims at addressing this problem by constructing personalized BP models from PPG signals. First, the PPG features that can well reflect the changes of individual BP with the physiological state were extracted. Afterwards, an error feedback incremental support vector regression (EFISVR) model was designed to achieve high-accuracy BP measurement of a subject, which can quickly be adapted to new samples without retraining the whole model. Results show that the constructed model can accurately predict the BP values of a subject for at least three months. The mean absolute error (MAE) of BP estimation were 3.11 mmHg for systolic BP (SBP) and 2.47 mmHg for diastolic BP (DBP). The proposed EFISVR model is lightweight which can be integrated into wearables and other edge devices, as a part of Internet of Things (IoT) applications. The advantages of lightweight, few-shot learning and high precision make the model suitable for applications in real-life scenarios.

The infant sleep-wake behavior is an essential indicator of physiological and neurological system maturity, the circadian transition of which is important for evaluating the recovery of preterm infants from inadequate physiological function and cognitive disorders. Recently, camera-based infant sleep-wake monitoring has been investigated, but the challenges of generalization caused by variance in infants and clinical environments are not addressed for this application. In this paper, we conducted a multi-center clinical trial at four hospitals to improve the generalization of camera-based infant sleep-wake monitoring. Using the face videos of 64 term and 39 preterm infants recorded in NICUs, we proposed a novel sleep-wake classification strategy, called consistent deep representation constraint (CDRC), that forces the convolutional neural network (CNN) to make consistent predictions for the samples from different conditions but with the same label, to address the variances caused by infants and environments. The clinical validation shows that by using CDRC, all CNN backbones obtain over 85% accuracy, sensitivity, and specificity in both the cross-age and cross-environment experiments, improving the ones without CDRC by almost 15% in all metrics. This demonstrates that by improving the consistency of the deep representation of samples with the same state, we can significantly improve the generalization of infant sleep-wake classification.

Heart rate variability (HRV) is a crucial metric that quantifies the variation between consecutive heartbeats, serving as a significant indicator of autonomic nervous system (ANS) activity. It has found widespread applications in clinical diagnosis, treatment, and prevention of cardiovascular diseases. In this study, we proposed an optical model for defocused speckle imaging, to simultaneously incorporate out-of-plane translation and rotation-induced motion for highly-sensitive non-contact seismocardiogram (SCG) measurement. Using electrocardiogram (ECG) signals as the gold standard, we evaluated the performance of photoplethysmogram (PPG) signals and speckle-based SCG signals in assessing HRV. The results indicated that the HRV parameters measured from SCG signals extracted from laser speckle videos showed higher consistency with the results obtained from the ECG signals compared to PPG signals. Additionally, we confirmed that even when clothing obstructed the measurement site, the efficacy of SCG signals extracted from the motion of laser speckle patterns persisted in assessing the HRV levels. This demonstrates the robustness of camera-based non-contact SCG in monitoring HRV, highlighting its potential as a reliable, non-contact alternative to traditional contact-PPG sensors.

Living-skin detection is an important step for imaging photoplethysmography and biometric anti-spoofing. In this paper, we propose a new approach that exploits spatio-temporal characteristics of structured light patterns projected on the skin surface for living-skin detection. We observed that due to the interactions between laser photons and tissues inside a multi-layer skin structure, the frequency-domain sharpness feature of laser spots on skin and non-skin surfaces exhibits clear difference. Additionally, the subtle physiological motion of living-skin causes laser interference, leading to brightness fluctuations of laser spots projected on the skin surface. Based on these two observations, we designed a new living-skin detection algorithm to distinguish skin from non-skin using spatio-temporal features of structured laser spots. Experiments in the dark chamber and Neonatal Intensive Care Unit (NICU) demonstrated that the proposed setup and method performed well, achieving a precision of 85.32%, recall of 83.87%, and F1-score of 83.03% averaged over these two scenes. Compared to the approach that only leverages the property of multilayer skin structure, the hybrid approach obtains an averaged improvement of 8.18% in precision, 3.93% in recall, and 8.64% in F1-score. These results validate the efficacy of using frequency domain sharpness and brightness fluctuations to augment the features of living-skin tissues irradiated by structured light, providing a solid basis for structured light based physiological imaging.

At Present, cameras are increasingly used to measure physiological signals from human face and body for contactless health monitoring, thereby eliminating mechanical contact with the skin that are common in wearable sensors. This is an emerging research direction developing rapidly in the last decade and which is now gradually maturing into products for patient monitoring. Advancements in biomedical optics, physiological measurement, computer vision and artificial intelligence (AI) enabled various camera-based measurements, including vital signs like heart rate (HR), respiration rate (RR), oxygen saturation (SpO2), blood pressure (BP), and physiological markers that have diagnostic capabilities, such as the detection of arrhythmia, atrial fibrillation, apnea, hypertension, etc. Image and video analysis also permit the measurement of human semantics, context and behaviours that provide new insights into health informatics (e.g., facial analysis and body actigraphy for the assessment of patient delirium), which is a unique advantage of camera sensors as compared to biomedical sensors, like e.g., photoplethysmography (PPG) and electrocardiogram (ECG). Camera-based health monitoring will bring a rich set of compelling healthcare applications that directly improve upon contact-based monitoring solutions in various scenarios like clinical units including e.g., the intensive care unit (ICU), the neonatal ICU (NICU) or sleep centers, and assisted-living homes (e.g., elderly homes or confinement centers), improving patient care experience and people's quality of life.

Infant cry is a crucial indicator that offers valuable insights into their physical and mental conditions, such as hunger and pain. However, the scarcity of infant cry datasets hinders the model's generalization in real-life scenarios. The varying voiceprint characteristics among infants further exacerbate this challenge, deteriorating the model's performance on unseen infants. To this end, we propose a multi-task model for Infant Cry Detection and Reasoning (ICDR). It leverages datasets from two tasks to enrich data diversity and introduces an efficient attention module to achieve inter-task feature supplementarity. To mitigate the impact of subject differences, ICDR introduces an intra-task contrastive mixture of experts (CMoE) module that adaptively allocates experts to reduce subject variance and applies contrastive learning to enhance the representation consistency of samples from different infants in the same state. Extensive cross-subject experiments show that ICDR outperforms the state-of-the-art models in infant cry detection and reasoning, with an improvement of 2-9% in the F1-score. This demonstrates that multi-task learning effectively enhances the model's generalization ability by inter-task attention and intra-task CMoE.

Sleep monitoring plays a vital role in various scenarios such as hospitals and living-assisted homes, contributing to the prevention of sleep accidents as well as the assessment of sleep health. Contactless camera-based sleep monitoring is promising due to its user-friendly nature and rich visual semantics. However, the privacy concern of video cameras limits their applications in sleep monitoring. In this paper, we explored the opportunity of using a defocused camera that does not allow identification of the monitored subject when measuring sleep-related parameters, as face detection and recognition are impossible on optically blurred images. We proposed a novel privacy-protected sleep parameters measurement framework, including a physiological measurement branch and a semantic analysis branch based on ResNet-18. Four important sleep parameters are measured: heart rate (HR), respiration rate (RR), sleep posture, and movement. The results of HR, RR, and movement have strong correlations with the reference (HR: R = 0.9076; RR: R = 0.9734; Movement: R = 0.9946). The overall mean absolute errors (MAE) for HR and RR are 5.2 bpm and 1.5 bpm respectively. The measurement of HR and RR achieve reliable estimation coverage of 72.1% and 93.6%, respectively. The sleep posture detection achieves an overall accuracy of 94.5%. Experimental results show that the defocused camera is promising for sleep monitoring as it fundamentally eliminates the privacy issue while still allowing the measurement of multiple parameters that are essential for sleep health informatics.

Contact-free sensing gained much traction in the past decade. While remote monitoring of some parameters (heart rate) is approaching clinical levels of precision, others remain challenging (blood pressure). We investigated the feasibility of estimating blood pressure (BP) via pulse transit time (PTT) in a novel remote single-site manner, using a modified RGB camera. A narrow-band triple band-pass filter allowed us to measure the PTT between different skin layers, harvesting information from green and near-infrared wavelengths. The filter minimizes the inter-channel influence and band overlap, however, some overlap remains within the filter bands. We further resolve this using a color-channel model and a novel channel-separation method. Using the proposed setup and algorithm, we obtained multi-wavelength (MW) PTTs in an experiment inducing BP changes to 9 subjects. The results showed good absolute Pearson’s correlation coefficient between both MW PTT and systolic BP (R = 0.61, p = 0.08) as well as diastolic BP (R = 0.54, p = 0.05), pointing to feasibility of the proposed novel remote MW BP estimation via PTT. This was further confirmed in a leave-one-subject-out experiment, where a simple Random Forest regression model achieved mean absolute errors of 3.59 and 2.63 mmHg for systolic and diastolic BP respectively.

Blood pressure (BP) monitoring has become a daily necessity for well-being management. Camera-based BP monitoring has attracted much attention due to its comfort and convenience. It extracts photoplethysmographic (PPG) signals from the facial skin remotely and calculates morphological features for BP estimation. Although green light with strong pulsatility is commonly used, its penetration depth in skin tissues is shorter than that of the near infrared (NIR) light, indicating that the NIR light may contain more distinct PPG features (i.e., dicrotic wave) that could be more beneficial for BP estimation. In this study, we investigated the wavelength-dependency of the camera-PPG waveform for BP measurement. In particular, two morphological features (K-value and Augmentation Index) from the narrow-band green (550nm), red (660nm), and NIR (850nm) channels were measured and compared for BP calibration. Experimental data were collected from 20 healthy adult subjects using the ice water stimulation protocol. The results show that the camera-PPG waveform features have clear wavelength-dependency. The NIR channel that has deeper skin penetrability (than the green channel) but higher pulsatility (than the red channel) gives more descriptive morphological features related to BP. The conclusions drawn from this study inspire the selection of wavelength for PPG morphologv-based BP measurement.

Infants need adequate sleep to develop their brain and cardiovascular systems, especially for preterm infants in the Neonatal Intensive Care Unit. Camera-based infant monitoring is an emerging direction of research in video health monitoring. Intuitively, camera can easily identify the sleep-awake stage of infants by detecting the state of eyes, e.g. closed or opening. Thus in this paper, we propose to explore the unique advantage of camera-based facial analysis for sleep-awake detection, as a fundamental step toward infant sleep monitoring. A multi-center clinical trial was conducted to collect infant videos for investigating the feasibility of our proposal. A benchmark including four machine learning methods of SVM, KNN, MLP, and CNN (ResNet18) was set up to classify the sleep/awake stage of infants. To alleviate the overfitting issue caused by over-sampling of a sleeping infant, we propose to integrate ResNet18 with the contrastive learning strategy to strengthen the consistency of facial features learned from different infants. The clinical evaluation shows that all benchmarked methods obtained an accuracy above 75% while the proposed method achieved the best accuracy of 86%. This invokes further explorations of using facial/eye features of infants for sleep-awake staging, towards intelligent contactless sleep analysis of infants in combination with camera-based vital signs monitoring.

Advancements in video health monitoring opened up new opportunities for contactless infant monitoring in the Neonatal Intensive Care Unit (NICU). Several studies have demonstrated the potential of heart rate variability (HRV) as an early warning measure for acute illnesses, such as late-onset sepsis and bradycardia. Conventional HRV analysis typically relies on electrocardiography (ECG) or contact photoplethysmography (contact-PPG) that may damage infants’ fragile skin. This study aims to investigate the feasibility of camera-PPG based HRV measurement for preterm infants in NICU, i.e. an important vital sign that was less studied in this scenario. A clinical study was conducted in two hospitals involving 41 infants, focused on comparing the HRV parameters obtained from camera-PPG and contact-PPG with the reference of ECG. Results show that camera-HRV has similar performance as contact-HRV in cases without motion distortions, and it has smaller errors and better robustness in the presence of motion distortions. Strong correlations can be observed between camera-PPG, contactPPG and ECG in several HRV parameters (Mean IBI, SDNN, LF, VLF, and SD2), though the performance on short-term variability parameters need to be improved further. The findings suggest that camera-PPG could be a promising tool for motionrobust HRV estimation in NICU, offering a potential surrogate for contact-PPG in early-warning of deteriorations of preterm infants.

The monitoring of vital signs such as heart rate (HR) and respiratory rate (RR) during sleep is important for the assessment of sleep quality and detection of sleep disorders. Camera-based HR and RR monitoring gained popularity in sleep monitoring in recent years. However, they are all facing with serious privacy issues when using a video camera in the sleeping scenario. In this paper, we propose to use the defocused camera to measure vital signs from optically blurred images, which can fundamentally eliminate the privacy invasion as face is difficult to be identified in obtained blurry images. A spatial-redundant framework involving living-skin detection is used to extract HR and RR from the defocused camera in NIR, and a motion metric is designed to exclude outliers caused by body motions. In the benchmark, the overall Mean Absolute Error (MAE) for HR measurement is 4.4 bpm, for RR measurement is 5.9 bpm. Both have quality drops as compared to the measurement using a focused camera, but the degradation in HR is much less, i.e. HR measurement has strong correlation with the reference (R ≥ 0.90). Preliminary experiments suggest that it is feasible to use a defocused camera for cardio-respiratory monitoring while protecting the privacy. Further improvement is needed for robust RR measurement, such as by PPG-modulation based RR extraction.

The monitoring of hand hygiene activities can effectively reduce infection and contamination in the Intensive Care Unit (ICU). In this paper, we created a clinical dataset using CCTV cameras installed in ICU to explore the feasibility of recognizing the hand-washing steps of clinicians. A video processing architecture including hand landmark detection and classification is presented. Such a system can potentially be used to alarm clinicians to follow the guidelines of hand hygiene. The experimental results show that the average accuracy of our methodology can achieve 95% under the personalized model and 56% under the generalized model. The preliminary results suggest that hand hygiene is subject-dependent, which is related to individuals’ palm size and washing habits. The cross-subject modeling or subject-adaptive learning can be applied to further improve the accuracy of recognition towards a more generalized solution. The insights of the study are helpful for designing a hand hygiene scoring and alarming system as a part of hospital IoT. The hospital data and code are available at https://github.com/SunnySideUp11/Hand-Hygiene-ICU.

Lung sound analysis has been used for assessing the lung conditions of children with community-acquired pneumonia (CAP). However, the inconsistent data distribution, caused by variant CAP symptoms appearing at different ages of children, limits the generalization ability of most diagnostic models. The data scarcity will further exacerbate this problem. Therefore, we propose a contrastive embedding-based domain adaptation network (CEDANN) to eliminate individual differences and alleviate data scarcity for improving the generalization ability. It forces the embedding layer to learn the subject-independent but task-dependent features by the adversarial learning between the domain classifier and the task classifier. Multiple contrast input tuples are constructed by combining samples from different classes to increase input combinations to alleviate data scarcity. The proposed method is evaluated on a multi-center clinical dataset. The subject-independent experiments show that CEDANN improves the sensitivity from 45.93% to 59.06% and the specificity from 35.07% to 59.55% in identifying CAP-confirmed, symptomatic relief, and recovery children. It demonstrates the effectiveness of CEDANN in the diagnosis and prognosis of children CAP.

Remote photoplethysmography (rPPG) can be used to monitor driver’s cardio-respiratory functions in automotive for improving the safety of driving. To understand the challenges of rPPG in this application, we created a benchmark of latest rPPG algorithms based on the MR-NIRP Car dataset, selecting the representative methods from both the physiological model based (PBV and DIS) and deep learning based (Supervised Learning and Contrastive Learning) approaches. The experimental results show that the physiological model based methods are generally more robust in this challenging scenario with vigorous motions and dynamic lighting changes, typically DIS outperforms others, with an average MAE of 6.5 bpm on RGB videos and 15.9 bpm on NIR videos. The benchmark indicates that upgrading the single wavelength NIR setup to multi-wavelength is the essential step towards robust heart-rate monitoring in automotive.

The monitoring of epilepsy patients in non-hospital environment is highly desirable, where ultra-low power wearable seizure detection devices are essential in such a system. The state-of-the-art epileptic seizure detection algorithms targeting such devices either rely on manual feature extractions, which can be biased due to the experience of experts, or deep neural networks, which suffer from high computation complexity. In this paper, we propose a lightweight deep learning model, LightSeizureNet (LSN), for real-time epileptic seizure detection based on raw EEG data in ultra-low power wearable seizure detection devices. The proposed LSN model includes a patient-independent version and a patient-specific version, both of which avoids manual feature extractions and high computation complexity, while maintaining good classification accuracy. Dilated one-dimensional (1D) convolution, global average pooling, and kernel-wise pruning are adopted to compress the LSN model. The proposed models are evaluated on the CHB-MIT scalp EEG database. The patient-independent LSN model achieves 97.09% accuracy with 6.2M MACs, while the patient-specific LSN model achieves 99.77% accuracy with 3.7 M MACs, which are competitive compared to the state of the art in terms of accuracy and complexity. Furthermore, the proposed model is highly interpretable, which is missing in many previous works. By using a uniform approach to explore the interpretability of the proposed model, fine-grained information such as the activated brain region and the frequency of brainwave during seizures is obtained for clinical diagnosis.

Body weight, as one of the biometric traits, has been studied in both the forensic and medical domains. However, estimating weight directly from 2-D images is particularly challenging since visual inspection is rather sensitive to the distance between the subject and camera, even for frontal view images. In this case, the widely used body mass index (BMI), which is associated with body height and weight, can be employed as a measure of weight to indicate health conditions. Previous works on the estimation of BMI have predominantly focused on using multiple 2-D images, 3-D images, or facial images; however, these cues are not always available. To address this issue, we explore the feasibility of obtaining BMI from a single 2-D body image with the dual-branch regression framework proposed in this work. More specifically, the framework comprises an anthropometric feature computation branch and a deep learning-based feature extraction branch. One aggregation layer maps all the features to an estimated BMI value. In addition, a new public 2-D image-to-BMI dataset, which contains 4189 images (1477 males and 2712 females) from approximately 3000 subjects with attributes including gender, age, height, and weight, was collected and released to facilitate the study. Extensive experiments confirm that the proposed framework combining anthropometric features and deep features outperforms the single-type feature approaches to BMI estimation in most cases.

In Magnetic Resonance Imaging (MRI), cardiac triggering that synchronizes data acquisition with cardiac contractions is an essential technique for acquiring high-quality images. Triggering is typically based on the Electrocardiogram (ECG) signal (e.g. R-peak). Since ECG acquisition involves extra workflow steps like electrode placement and ECG signals are usually disturbed by magnetic fields in high Magnetic Resonance (MR) systems, we explored camera-based photoplethysmography (PPG) as an alternative. We used the in-bore camera of a clinical MR system to investigate the feasibility and challenges of camera-based cardiac triggering. Data from ECG, finger oximeter and camera were synchronously collected. Compared to finger-PPG, camera-based PPG provides a higher availability of the signal and the PPG marker delay relative to the ECG R-peak is considerably less with a camera monitoring the forehead. The insights obtained in this study provide a basis for an envisioned system-design phase.

The papers in this special section focus on camera-based monitoring for pervasive healthcare informatics. Measuring physiological signals from the human face and body using video cameras is an emerging research topic that has grown rapidly in the last decade. Remote cameras (in both visible and infrared wavelengths) can be used to measure vital signs from a human body based on skin optics or body movements thereby avoiding mechanical contact with the skin. Camera-based health monitoring will bring a rich set of compelling healthcare applications that directly improve upon contact-based monitoring solutions and impact people’s care experience and quality of life in various scenarios, such as in hospital care units, sleep/senior centers, assisted-living homes, telemedicine and e-health, baby/elderly care at home, fitness and sports, driver monitoring in automotive applications, cardiac/ respiratory gating for MRI/CT, AR/VR based therapy and clinical training, e

The time interval between consecutive heartbeats (interpulse interval, IPI) has previously been suggested for securing mobile-health solutions. This time interval is known to contain a degree of randomness, permitting the generation of a time- and person-specific identifier. It is commonly assumed that only devices trusted by a person can make physical contact with him/her, and that this physical contact allows each device to generate a similar identifier based on its own cardiac recordings. Under these conditions, the identifiers generated by different trusted devices can facilitate secure authentication. Recently, a wide range of techniques have been proposed for measuring heartbeats remotely, a prominent example of which is remote photoplethysmography (rPPG). These techniques may pose a significant threat to heartbeat-based security, as an adversary may pretend to be a trusted device by generating a similar identifier without physical contact, thus bypassing one of the core security conditions. In this paper, we assess the feasibility of such remote attacks using state-of-the-art rPPG methods. Our evaluation shows that rPPG has similar accuracy as contact PPG and, thus, forms a substantial threat to heartbeat-based-security systems that permit trusted devices to obtain their identifiers from contact PPG recordings. Conversely, rPPG cannot obtain an accurate representation of an identifier generated from electrical cardiac signals, making the latter invulnerable to state-of-the-art remote attacks.

This paper introduces a powerful filtering method that exploits the physiological and optical properties of skin reflections to improve the performance of remote photoplethysmography (rPPG). Based on the fact that the pulsatile and nonpulsatile (e.g., intensity and specular changes) components have different reflection-spectra in a multi-wavelength camera, we propose to use their different characteristic color changes as a soft criterion to filter the RGB-signals in the frequency domain, such that the AC-components containing clear color distortions can be suppressed before the actual pulse extraction. This leads to a novel “Color-Distortion Filter” (CDF) that can be used as a common pre-processing step for arbitrary rPPG algorithms to increase their robustness. The benchmark in challenging fitness recordings shows that CDF brings significant and consistent improvements to all benchmarked rPPG algorithms, and drives all multi-channel approaches to a similar high quality-level.

Camera-based remote pulse rate monitoring can be used during fitness exercise to optimize the effectiveness of a workout. However, such monitoring suffers from vigorous body motions and dynamic illumination changes due to exercise, which may lead to erroneous estimates. To better cope with this, we propose a quality metric, comprised of a front-end metric and a back-end metric, to indicate the monitoring conditions (e.g. luminance, skin property) and assess the reliability of pulse rate measurement (e.g. signal quality). The proposed quality metric has been thoroughly benchmarked on 78 videos recorded in a fitness setting. The experimental results show that (i) appropriate light source intensity variation and its angle variation in the front-end metric are critical indicators for pulse rate measurement accuracy, and (ii) the back-end metric can effectively indicate/reject unreliable estimates. The proposed method in this paper is the first quality metric for camera-based pulse rate monitoring, validated for the challenging use-case of fitness exercises.