Existing respiratory monitoring techniques primarily focus on respiratory rate measurement, neglecting the potential of using thoracoabdominal patterns of respiration for infant lung health assessment. To bridge this gap, we exploit the unique advantage of spatial redundancy of a camera sensor to analyze the infant thoracoabdominal respiratory motion. Specifically, we propose a camera-based respiratory imaging (CRI) system that utilizes optical flow to construct a spatio-temporal respiratory imager for comparing the infant chest and abdominal respiratory motion, and employs deep learning algorithms to identify infant abdominal, thoracoabdominal synchronous, and thoracoabdominal asynchronous patterns of respiration. To alleviate the challenges posed by limited clinical training data and subject variability, we introduce a novel multiple-expert contrastive learning (MECL) strategy to CRI. It enriches training samples by reversing and pairing different-class data, and promotes the representation consistency of same-class data through multi-expert collaborative optimization. Clinical validation involving 44 infants shows that MECL achieves 70% in sensitivity and 80.21% in specificity, which validates the feasibility of CRI for respiratory pattern recognition. This work investigates a novel video-based approach for assessing the infant thoracoabdominal patterns of respiration, revealing a new value stream of video health monitoring in neonatal care.

Current camera-based infant monitoring mainly focuses on physiological measurement, overlooking its important semantic analysis potential for detecting accidental suffocation caused by oronasal occlusion during sleep. However, developing a robust infant suffocation risk detection model typically requires substantial labeled data, which is very difficult to obtain in real-world scenarios. To address this, we utilized the text-to-image diffusion model to generate diverse infant images depicting oronasal occlusion and non-occlusion scenarios controlled by text prompts. To ease the process of labeling, self- and semi-supervised learning algorithms are leveraged to learn the semantic information from unlabeled data with the support of minimal labeled data to train different model architectures. To evaluate the feasibility of this solution, we conducted a clinical trial in the neonatology department, which collected video data from 22 infants under various oronasal occlusion scenarios using breathable covers (e.g. clinical tissue). The clinical evaluation shows that most models trained on 25,000 generated images achieved over 90% performance on metrics of accuracy, recall, and F1-score, outperforming conventional approaches that pre-train and fine-tune the model using over 90,000 labeled task-related online images. This demonstrates the feasibility of leveraging text-to-image generated data to achieve robust camera-based infant suffocation risk detection, so as to secure the sleep safety of infants. More importantly, it beacons the potential of using text-based large-scale model to solve the general issue of scarcity of human data in artificial intelligence-based healthcare or clinical applications.

Respiratory spatial patterns describe the distribution and dynamics of lung conditions, and monitoring of their asymmetry or irregularities enables a more comprehensive assessment of respiratory functions. The feasibility of using the camera pixel array sensing combined with machine learning for analyzing respiratory spatial patterns was demonstrated, however, this approach faces challenges in patient generalization due to limited clinical data and individual respiratory variability. Data augmentation methods may address this by synthesizing new data, but they have a risk of destroying the symmetry or regular semantic information of respiratory patterns. To address this, we propose a prototype-driven hard-sample contrastive learning (PHCL) method tailored for camera-based respiratory imaging analysis. It first separates the samples into simple and hard-to-learn samples using prototypes and Gini-index distance measurement. Then it synthesizes a new feature by blending one-class simple samples and other-class hard samples to construct a transition boundary between different classes, so as to broaden the feature distribution. Then it employs contrastive learning to emphasize feature consistency between prototypes and hard same-class samples from different subjects to mitigate individual respiratory variability and refine class boundaries. Extensive experiments were conducted in the neonatal intensive care unit and the thoracic surgery department, where PHCL outperforms image augmentation and advanced feature augmentation methods by 1-10% in both accuracy and F1-score. Our work provides valuable insights into the analysis of asymmetric and irregular respiratory activities.

Conventional physiological monitoring relies on multiple contact-based biomedical sensors, such as electrocardiograms, pulse oximeters, and blood pressure (BP) cuffs, which often necessitate the use of cumbersome sensors (e.g., electrodes, patches, diodes) and extensive wiring. These conventional methods not only impose significant inconvenience on both caregivers and patients but also elevate the risk of patient infections. In this study, we introduce a novel non-contact, multi-parameter vital signs monitoring platform based on the red, green and near Infrared (RG-IR) spectral imaging system, which measures five critical patient parameters in the Intensive Care Unit (ICU), including heart rate (HR), heart rate variability (HRV), respiratory rate (RR), blood oxygen saturation (SpO$_{2}$) and BP. Clinical trials were conducted in the ICUs of two hospitals, involving 30 critically ill patients. The clinical outcomes indicate that our system achieves reasonable performance, with mean absolute errors of 1.89 bpm for HR, 19.04 ms for SDNN (standard deviation of normal-to-normal intervals), 0.99 bpm for RR, 2.75% for SpO$_{2}$ and 5.67-8.98 mmHg for BP, providing timely physiological information and marking a significant step toward contactless ICU monitoring. Additionally, the real-time performance of the core algorithms were validated on low-cost embedded chips, demonstrating the system's capability for edge computation and on-board patient monitoring. The proposed system has entered the registration process of the National Medical Products Administration (NMPA) as a Class II medical device (No. 2024007).

Heart rate variability (HRV) can be used as an effective indicator for the maturity of the neonatal autonomic nervous system (ANS) and some acute neonatal diseases (e.g., respiratory distress syndrome and late-onset sepsis). Advances in contactless sensing technology have led to the rapid development of camera-based physiological monitoring, but few studies have focused on HRV measurement in the Neonatal Intensive Care Unit. This study aims to evaluate the performance of camera-based neonatal HRV monitoring (camera-HRV) using camera–photoplethysmography (PPG), and to further investigate the feasibility of applying camera-HRV for tracking ANS maturation through quantitative analysis of 12 HRV indices on quiet infants with limited motions. The clinical experimental results show that as the ANS of newborns matures, the trends of camera-HRV indices are consistent with those obtained by electrocardiogram (ECG) and PPG. Furthermore, the performance of camera-HRV in predicting the postmenstrual age of newborns is comparable to that obtained by ECG and PPG, which can predict 63.89% of infants with an accuracy of less than ±2 weeks. In particular, camera-HRV shows excellent performance in the classification of preterm/term infants, with an optimal accuracy of 0.90 and an F1 score of 0.88. The pilot clinical study suggests that camera-HRV can be a feasible tool for tracking the development of neonatal ANS.

The infant sleep-wake behavior is an essential indicator of physiological and neurological system maturity, the circadian transition of which is important for evaluating the recovery of preterm infants from inadequate physiological function and cognitive disorders. Recently, camera-based infant sleep-wake monitoring has been investigated, but the challenges of generalization caused by variance in infants and clinical environments are not addressed for this application. In this paper, we conducted a multi-center clinical trial at four hospitals to improve the generalization of camera-based infant sleep-wake monitoring. Using the face videos of 64 term and 39 preterm infants recorded in NICUs, we proposed a novel sleep-wake classification strategy, called consistent deep representation constraint (CDRC), that forces the convolutional neural network (CNN) to make consistent predictions for the samples from different conditions but with the same label, to address the variances caused by infants and environments. The clinical validation shows that by using CDRC, all CNN backbones obtain over 85% accuracy, sensitivity, and specificity in both the cross-age and cross-environment experiments, improving the ones without CDRC by almost 15% in all metrics. This demonstrates that by improving the consistency of the deep representation of samples with the same state, we can significantly improve the generalization of infant sleep-wake classification.

Living-skin detection is an important step for imaging photoplethysmography and biometric anti-spoofing. In this paper, we propose a new approach that exploits spatio-temporal characteristics of structured light patterns projected on the skin surface for living-skin detection. We observed that due to the interactions between laser photons and tissues inside a multi-layer skin structure, the frequency-domain sharpness feature of laser spots on skin and non-skin surfaces exhibits clear difference. Additionally, the subtle physiological motion of living-skin causes laser interference, leading to brightness fluctuations of laser spots projected on the skin surface. Based on these two observations, we designed a new living-skin detection algorithm to distinguish skin from non-skin using spatio-temporal features of structured laser spots. Experiments in the dark chamber and Neonatal Intensive Care Unit (NICU) demonstrated that the proposed setup and method performed well, achieving a precision of 85.32%, recall of 83.87%, and F1-score of 83.03% averaged over these two scenes. Compared to the approach that only leverages the property of multilayer skin structure, the hybrid approach obtains an averaged improvement of 8.18% in precision, 3.93% in recall, and 8.64% in F1-score. These results validate the efficacy of using frequency domain sharpness and brightness fluctuations to augment the features of living-skin tissues irradiated by structured light, providing a solid basis for structured light based physiological imaging.

Infants need adequate sleep to develop their brain and cardiovascular systems, especially for preterm infants in the Neonatal Intensive Care Unit. Camera-based infant monitoring is an emerging direction of research in video health monitoring. Intuitively, camera can easily identify the sleep-awake stage of infants by detecting the state of eyes, e.g. closed or opening. Thus in this paper, we propose to explore the unique advantage of camera-based facial analysis for sleep-awake detection, as a fundamental step toward infant sleep monitoring. A multi-center clinical trial was conducted to collect infant videos for investigating the feasibility of our proposal. A benchmark including four machine learning methods of SVM, KNN, MLP, and CNN (ResNet18) was set up to classify the sleep/awake stage of infants. To alleviate the overfitting issue caused by over-sampling of a sleeping infant, we propose to integrate ResNet18 with the contrastive learning strategy to strengthen the consistency of facial features learned from different infants. The clinical evaluation shows that all benchmarked methods obtained an accuracy above 75% while the proposed method achieved the best accuracy of 86%. This invokes further explorations of using facial/eye features of infants for sleep-awake staging, towards intelligent contactless sleep analysis of infants in combination with camera-based vital signs monitoring.

Advancements in video health monitoring opened up new opportunities for contactless infant monitoring in the Neonatal Intensive Care Unit (NICU). Several studies have demonstrated the potential of heart rate variability (HRV) as an early warning measure for acute illnesses, such as late-onset sepsis and bradycardia. Conventional HRV analysis typically relies on electrocardiography (ECG) or contact photoplethysmography (contact-PPG) that may damage infants’ fragile skin. This study aims to investigate the feasibility of camera-PPG based HRV measurement for preterm infants in NICU, i.e. an important vital sign that was less studied in this scenario. A clinical study was conducted in two hospitals involving 41 infants, focused on comparing the HRV parameters obtained from camera-PPG and contact-PPG with the reference of ECG. Results show that camera-HRV has similar performance as contact-HRV in cases without motion distortions, and it has smaller errors and better robustness in the presence of motion distortions. Strong correlations can be observed between camera-PPG, contactPPG and ECG in several HRV parameters (Mean IBI, SDNN, LF, VLF, and SD2), though the performance on short-term variability parameters need to be improved further. The findings suggest that camera-PPG could be a promising tool for motionrobust HRV estimation in NICU, offering a potential surrogate for contact-PPG in early-warning of deteriorations of preterm infants.