Pulmonary Arterial Wedge Pressure (PAWP) is an essential cardiovascular hemodynamics marker to detect heart failure. In clinical practice, Right Heart Catheterization is considered a gold standard for assessing cardiac hemodynamics while non-invasive methods are often needed to screen high-risk patients from a large population. In this paper, we propose a multimodal learning pipeline to predict PAWP marker. We utilize complementary information from Cardiac Magnetic Resonance Imaging (CMR) scans (short-axis and four-chamber) and Electronic Health Records (EHRs). We extract spatio-temporal features from CMR scans using tensor-based learning. We propose a graph attention network to select important EHR features for prediction, where we model subjects as graph nodes and feature relationships as graph edges using the attention mechanism. We design four feature fusion strategies: early, intermediate, late, and hybrid fusion. With a linear classifier and linear fusion strategies, our pipeline is interpretable. We validate our pipeline on a large dataset of ${2},{641}$ subjects from our ASPIRE registry. The comparative study against state-of-the-art methods confirms the superiority of our pipeline. The decision curve analysis further validates that our pipeline can be applied to screen a large population. The code is available at https://github.com/prasunc/hemodynamics.

Right Heart Catheterization is a gold standard procedure for diagnosing Pulmonary Hypertension by measuring mean Pulmonary Artery Pressure (mPAP). It is invasive, costly, time-consuming and carries risks. In this paper, for the first time, we explore the estimation of mPAP from videos of noninvasive Cardiac Magnetic Resonance Imaging. To enhance the predictive capabilities of Deep Learning models used for this task, we introduce an additional modality in the form of demographic features and clinical measurements. Inspired by all-Multilayer Perceptron architectures, we present TabMixer, a novel module enabling the integration of imaging and tabular data through spatial, temporal and channel mixing. Specifically, we present the first approach that utilizes Multilayer Perceptrons to interchange tabular information with imaging features in vision models. We test TabMixer for mPAP estimation and show that it enhances the performance of Convolutional Neural Networks, 3D-MLP and Vision Transformers while being competitive with previous modules for imaging and tabular data. Our approach has the potential to improve clinical processes involving both modalities, particularly in noninvasive mPAP estimation, thus, significantly enhancing the quality of life for individuals affected by Pulmonary Hypertension. We provide a source code for using TabMixer at https://github.com/SanoScience/TabMixer .

Pulmonary Arterial Hypertension (PAH) is a lifethreatening disorder. The prediction of mortality in PAH patients can play a crucial role in the clinical management of this disease. The prediction of mortality from one modality is a difficult task that may only provide limited performance. Therefore, we propose a multimodal learning approach in this work to predict one-year mortality in PAH patients. We have utilised three modalities, which include extracted numerical imaging features, echo report categorical features, and echo report text features from Electronic Health Records (EHRs) of patients. We have proposed a feature integration module to combine features from multiple modalities. The text features have been extracted from the echo reports using the Bidirectional Encoder Representations from Transformers (BERT). An attention mechanism and a weighted summation method are also adopted during the process of feature integration. We have performed different experiments to evaluate the performance of the proposed framework for mortality prediction. The experimental results indicate that we can achieve the best AUC score of 0.89 for predicting one-year mortality by combining all three modalities. The source code of this paper is available at https://github.com/Mdnaimulislam/MultimodalTab.