Cancer survival prediction requires exploiting related multimodal information (e.g., pathological, clinical and genomic features, etc.) and it is even more challenging in clinical practices due to the incompleteness of patient's multimodal data. Furthermore, existing methods lack sufficient intra- and inter-modal interactions, and suffer from significant performance degradation caused by missing modalities. This manuscript proposes a novel hybrid graph convolutional network, entitled HGCN, which is equipped with an online masked autoencoder paradigm for robust multimodal cancer survival prediction. Particularly, we pioneer modeling the patient's multimodal data into flexible and interpretable multimodal graphs with modality-specific preprocessing. HGCN integrates the advantages of graph convolutional networks (GCNs) and a hypergraph convolutional network (HCN) through node message passing and a hyperedge mixing mechanism to facilitate intra-modal and inter-modal interactions between multimodal graphs. With HGCN, the potential for multimodal data to create more reliable predictions of patient's survival risk is dramatically increased compared to prior methods. Most importantly, to compensate for missing patient modalities in clinical scenarios, we incorporated an online masked autoencoder paradigm into HGCN, which can effectively capture intrinsic dependence between modalities and seamlessly generate missing hyperedges for model inference. Extensive experiments and analysis on six cancer cohorts from TCGA show that our method significantly outperforms the state-of-the-arts in both complete and missing modal settings. Our codes are made available at https://github.com/lin-lcx/HGCN.

Current representation learning methods for whole slide image (WSI) with pyramidal resolutions are inherently homogeneous and flat, which cannot fully exploit the multiscale and heterogeneous diagnostic information of different structures for comprehensive analysis. This paper presents a novel graph neural network-based multiple instance learning framework (i.e., H^2-MIL) to learn hierarchical representation from a heterogeneous graph with different resolutions for WSI analysis. A heterogeneous graph with the “resolution” attribute is constructed to explicitly model the feature and spatial-scaling relationship of multi-resolution patches. We then design a novel resolution-aware attention convolution (RAConv) block to learn compact yet discriminative representation from the graph, which tackles the heterogeneity of node neighbors with different resolutions and yields more reliable message passing. More importantly, to explore the task-related structured information of WSI pyramid, we elaborately design a novel iterative hierarchical pooling (IHPool) module to progressively aggregate the heterogeneous graph based on scaling relationships of different nodes. We evaluated our method on two public WSI datasets from the TCGA project, i.e., esophageal cancer and kidney cancer. Experimental results show that our method clearly outperforms the state-of-the-art methods on both tumor typing and staging tasks.

Survival prediction of esophageal cancer is an essential task for doctors to make personalized cancer treatment plans. However, handcrafted features from medical images need prior medical knowledge, which is usually limited and not complete, yielding unsatisfying survival predictions. To address these challenges, we propose a novel and efficient deep learning-based survival prediction framework for evaluating clinical outcomes before concurrent chemoradiotherapy. The proposed model consists of two key components: a 3D Coordinate Attention Convolutional Autoencoder (CACA) and an uncertainty-based jointly Optimizing Cox Model (UOCM). The CACA is built upon an autoencoder structure with 3D coordinate attention layers, capturing latent representations and encoding 3D spatial characteristics with precise positional information. Additionally, we designed an Uncertainty-based jointly Optimizing Cox Model, which jointly optimizes the CACA and survival prediction task. The survival prediction task models the interactions between a patient's feature signatures and clinical outcome to predict a reliable hazard ratio of patients. To verify the effectiveness of our model, we conducted extensive experiments on a dataset including computed tomography of 285 patients with esophageal cancer. Experimental results demonstrated that the proposed method achieved a C-index of 0.72, outperforming the state-of-the-art method.