Recent advancements in computer vision, including convolutional neural networks, multilayer perceptrons, graph-based methods and transformer architectures, have significantly improved image classification. However, applying these techniques to digital pathology, particularly gigapixel whole-slide images (WSIs), presents unique challenges due to their vast size and heterogeneity. We introduce FourierMIL, a multiple instance learning framework that leverages the discrete Fourier transform to efficiently capture global and local dependencies in WSIs. Unlike conventional approaches, FourierMIL is adaptable to diverse digital stains and pathology tasks. To evaluate its versatility, we tested FourierMIL on three distinct challenges using public and private datasets. (1) Metastasis detection in hematoxylin and eosin (H&E)- stained lymph node WSIs from CAncer MEtastases in LYmph nOdes challeNge (CAMELYON16) dataset. (2) Lung cancer classification (adenocarcinoma versus squamous cell carcinoma) using The Cancer Genome Atlas (TCGA) and the Clinical Proteomic Tumor Analysis Consortium (CPTAC) datasets. (3) Alzheimer's disease pathology identification in phospho-tau monoclonal antibody (AT8)- stained WSIs from the Understanding Neurologic Injury and Traumatic Encephalopathy (UNITE), the Framingham Heart Study (FHS), and the Boston University Alzheimer's Disease Research Center (ADC) cohorts. FourierMIL outperformed state-of-the-art methods across all tasks, demonstrating its robustness as an attention-free solution for diverse applications in digital pathology.

Multimodal machine learning models are being developed to analyze pathology images and other modalities, such as gene expression, to gain clinical and biological insights. However, most frameworks for multimodal data fusion do not fully account for the interactions between different modalities. Here, we present an attention-based fusion architecture that integrates a graph representation of pathology images with gene expression data and concomitantly learns from the fused information to predict patient-specific survival. In our approach, pathology images are represented as undirected graphs, and their embeddings are combined with embeddings of gene expression signatures using an attention mechanism to stratify tumors by patient survival. We show that our framework improves the survival prediction of human non-small cell lung cancers, outperforming existing state-of-the-art approaches that leverage multimodal data. Our framework can facilitate spatial molecular profiling to identify tumor heterogeneity using pathology images and gene expression data, complementing results obtained from more expensive spatial transcriptomic and proteomic technologies.

Deep learning is a powerful tool for whole slide image (WSI) analysis. Typically, when performing supervised deep learning, a WSI is divided into small patches, trained and the outcomes are aggregated to estimate disease grade. However, patch-based methods introduce label noise during training by assuming that each patch is independent with the same label as the WSI and neglect overall WSI-level information that is significant in disease grading. Here we present a Graph-Transformer (GT) that fuses a graph-based representation of an WSI and a vision transformer for processing pathology images, called GTP, to predict disease grade. We selected 4,818 WSIs from the Clinical Proteomic Tumor Analysis Consortium (CPTAC), the National Lung Screening Trial (NLST), and The Cancer Genome Atlas (TCGA), and used GTP to distinguish adenocarcinoma (LUAD) and squamous cell carcinoma (LSCC) from adjacent non-cancerous tissue (normal). First, using NLST data, we developed a contrastive learning framework to generate a feature extractor. This allowed us to compute feature vectors of individual WSI patches, which were used to represent the nodes of the graph followed by construction of the GTP framework. Our model trained on the CPTAC data achieved consistently high performance on three-label classification (normal versus LUAD versus LSCC: mean accuracy = 91.2 ± 2.5%) based on five-fold cross-validation, and mean accuracy = 82.3 ± 1.0% on external test data (TCGA). We also introduced a graph-based saliency mapping technique, called GraphCAM, that can identify regions that are highly associated with the class label. Our findings demonstrate GTP as an interpretable and effective deep learning framework for WSI-level classification.