Remote physiology, which involves monitoring vital signs without the need for physical contact, has great potential for various applications. Current remote physiology methods rely only on a single camera or radio frequency (RF) sensor to capture the microscopic signatures from vital movements. However, our study shows that fusing deep RGB and RF features from both sensor streams can further improve performance. Because these multimodal features are defined in distinct dimensions and have varying contextual importance, the main challenge in the fusion process lies in the effective alignment of them and adaptive integration of features under dynamic scenarios. To address this challenge, we propose a novel vital sensing model, named Fusion-Vital, that combines the RGB and RF modalities through the new introduction of pairwise input formats and transformer-based fusion strategies. We also perform comprehensive experiments based on a newly collected and released remote vital dataset comprising synchronized video-RF sensors, showing the superiority of the fusion approach over the previous single-sensor baselines in various aspects.

The noncontact respiration rate measurement (nRRM) method allows a system to monitor the breathing patterns of an individual without physical contact, which is crucial for regular health monitoring. Current nRRM approaches primarily depend on detecting minor variations in RGB profiles reflected from a camera to remotely extract respiration signals. However, these methods require continuous pixel-level tracking, which restricts their use on individuals in quasi-stationary sitting positions. To address this limitation, we propose a radiofrequency (RF)-Vital model, which leverages RF signals to extend the applicability of nRRM methods to individuals who exhibit global motions (GMs) and even walk around. The core idea of the RF-Vital model lies in the unique characteristics of RF signals: the RF signals received from a moving individual capture both their respiratory motions (RMs) and GMs through linear superposition while simultaneously providing the reflections of GM alone. To fully utilize such unique properties, we introduce a new RF modality that allows stable inclusion of micro-level respiration signatures, even when GMs are present. Additionally, we optimize the RF-Vital model using a novel multitask adversarial learning framework combined with a new loss function, which facilitates the direct mapping of the desired RMs as well as the self-supervised removal of GMs, thereby effectively filtering out RMs from mixtures of GMs and RMs. The proposed RFvital model was evaluated using newly published data sets. It demonstrated state-of-the-art performance in static conditions and achieved the significant milestone of enabling nRRM under moving conditions.