Gigapixel image analysis, particularly for whole slide images (WSIs), often relies on multiple instance learning (MIL). Under the paradigm of MIL, patch image representations are extracted and then fixed during the training of the MIL classifiers for efficiency consideration. However, the invariance of representations makes it difficult to perform data augmentation for WSI-level model training, which significantly limits the performance of the downstream WSI analysis. The current data augmentation methods for gigapixel images either introduce additional computational costs or result in a loss of semantic information, which is hard to meet the requirements for efficiency and stability needed for WSI model training. In this paper, we propose a Promptable Representation Distribution Learning framework (PRDL) for both patch-level representation learning and WSI-level data augmentation. Meanwhile, we explore the use of prompts to guide data augmentation in feature space, which achieves promptable data augmentation for training robust WSI-level models. The experimental results have demonstrated that the proposed method stably outperforms state-of-the-art methods.

Multimodal survival prediction combining genomic features with histopathology whole slide images (WSIs) has shown promise, but genomic data remains costly and inaccessible for widespread clinical adoption. In contrast, clinical data and multi-stained WSIs are routinely available. We present Contrastive Learning guided by Pathological Reports (CLPR), a multimodal survival prediction framework that integrates H&Estained WSIs, immunohistochemical (IHC)-stained WSIs, and structured pathological reports. Unlike genomic approaches, IHC provides protein-level tumor microenvironment information at lower cost, while pathological reports bridge clinical diagnosis with prognostic assessment. CLPR addresses key challenges in multimodal fusion through three innovations: (1) pathological report-guided contrastive learning that identifies semantically meaningful sample pairs for robust cross-modal alignment, overcoming unreliable attention scores under weak supervision; (2) a distribution-aware regularization constraint that explicitly preserves inter-group separability and intra-group compactness between risk cohorts, enhancing discriminative power; and (3) the first multimodal survival dataset combining H&E- and IHC-stained WSIs with structured pathological reports. On our clinical gastric cancer dataset (270 patients), CLPR achieves a C-index of 0.785 with significant risk stratification (log-rank p$C$-index: 0.772), demonstrating clinical versatility.

In the domain of histopathology analysis, existing representation learning methods for biomarkers prediction from whole slide images (WSIs) face challenges due to the complexity of tissue subtypes and label noise problems. This paper proposed a novel partial-label contrastive representation learning approach to enhance the discrimination of histopathology image representations for fine-grained biomarkers prediction. We designed a partial-label contrastive clustering (PLCC) module for partial-label disambiguation and a dynamic clustering algorithm to sample the most representative features of each category to the clustering queue during the contrastive learning process. We conducted comprehensive experiments on three gene mutation prediction datasets, including USTC-EGFR, BRCA-HER2, and TCGA-EGFR. The results show that our method outperforms 9 existing methods in terms of Accuracy, AUC, and F1 Score. Specifically, our method achieved an AUC of 0.950 in EGFR mutation subtyping of TCGA-EGFR and an AUC of 0.853 in HER2 0/1+/2+/3+ grading of BRCA-HER2, which demonstrates its superiority in fine-grained biomarkers prediction from histopathology whole slide images.

The development of computational pathology lies in the consensus that pathological characteristics of tumors are significant guidance for cancer diagnostics. Most existing research focuses on the inner-contextual information within each WSI yet ignores the possible inter-correlations between slides. As the development of tumors is a continuous process involving a series of histological, morphological, and genetic changes that accumulate over time, the similarities and differences between WSIs across various stages, grades, locations and patients should potentially contribute to the representation of WSIs and deserve to be taken into account in WSI modeling. To verify the advancement of introducing the slide inter-correlations into the representation learning of WSIs, we proposed a generic WSI analysis pipeline SlideGCD that can be adapted to any existing Multiple Instance Learning (MIL) frameworks and improve their performance. With the new paradigm, the prior knowledge of cancer development can participate in the end-to-end workflow, which concurrently initializes and refines the slide representation, as a guide for message passing in the slide-based graph. Extensive comparisons and experiments are conducted to validate the effectiveness and robustness of the proposed pipeline across 4 different tasks, including cancer subtyping, cancer staging, survival prediction, and gene mutation prediction, with 8 representative SOTA WSI analysis frameworks as backbones. The code is available at https://github.com/HFUT-miaLab/SlideGCD.

Multiple instance learning (MIL) based whole slide image (WSI) classification is often carried out on the representations of patches extracted from WSI with a pre-trained patch encoder. The performance of classification relies on both patch-level representation learning and MIL classifier training. Most MIL methods utilize a frozen model pre-trained on ImageNet or a model trained with self-supervised learning on histopathology image dataset to extract patch image representations and then fix these representations in the training of the MIL classifiers for efficiency consideration. However, the invariance of representations cannot meet the diversity requirement for training a robust MIL classifier, which has significantly limited the performance of the WSI classification. In this paper, we propose a Self-Supervised Representation Distribution Learning framework (SSRDL) for patch-level representation learning with an online representation sampling strategy (ORS) for both patch feature extraction and WSI-level data augmentation. The proposed method was evaluated on three datasets under three MIL frameworks. The experimental results have demonstrated that the proposed method achieves the best performance in histopathology image representation learning and data augmentation and outperforms state-of-the-art methods under different WSI classification frameworks. The code is available at https://github.com/lazytkm/SSRDL.

Large-scale pre-training models have promoted the development of histopathology image analysis. However, existing self-supervised methods for histopathology images primarily focus on learning patch features, while there is a notable gap in the availability of pre-training models specifically designed for WSI-level feature learning. In this paper, we propose a novel self-supervised learning framework for pan-cancer WSI-level representation pre-training with the designed position-aware masked autoencoder (PAMA). Meanwhile, we propose the position-aware cross-attention (PACA) module with a kernel reorientation (KRO) strategy and an anchor dropout (AD) mechanism. The KRO strategy can capture the complete semantic structure and eliminate ambiguity in WSIs, and the AD contributes to enhancing the robustness and generalization of the model. We evaluated our method on 7 large-scale datasets from multiple organs for pan-cancer classification tasks. The results have demonstrated the effectiveness and generalization of PAMA in discriminative WSI representation learning and pan-cancer WSI pre-training. The proposed method was also compared with 8 WSI analysis methods. The experimental results have indicated that our proposed PAMA is superior to the state-of-the-art methods. The code and checkpoints are available at https://github.com/WkEEn/PAMA.

Traditional PCR/NGS-based multigene panel testing is time-consuming and costly. Predicting EGFR mutations directly from H&E stained whole slide images (WSIs) can alleviate these limitations. Furthermore, histopathological reports contain valuable textual information that correlates with tissue areas in WSIs. However, recent research mainly analyses EGFR mutation status only from a single modality, ignoring rich information contained in reports. In this paper, we propose a report-guided cross-modal representation learning method for predicting EGFR mutations by WSIs. Specifically, we reconstruct report-level embeddings through exploring intrinsic relationships between diagnostic words in histopathological reports and tissue areas in WSIs. Finally, reconstructed histopathological report embedding and aggregated WSI embedding are fused for final prediction. More importantly, molecular testing report is also introduced as prior supervision information at the training stage to guarantee semantic consistency of fused feature and molecular report embedding. We evaluate our method on the TCGA-EGFR public benchmark dataset and an in-house clinical dataset (USTC-EGFR). Experimental results demonstrate that our method outperforms existing approaches in EGFR mutation prediction, highlighting the benefits of cross-modal learning in enhancing feature representational ability. The code is available at https://github.com/HFUT-miaLab/RCRL.

Forecasting human trajectory is an essential technology in intelligent surveillance systems, robot navigation systems, autonomous driving systems, etc. Most of the trajectory prediction models based on RNN and Transformers use autoregressive methods to generate future trajectories, which may accumulate displacement errors and are inefficient for training and testing. To address these problems, we propose a novel decoder named MRG decoder, which introduces a Mapping-Refinement-Generation structure to generate trajectory in a non-autoregressive manner. Furthermore, we design the MRGTraj trajectory prediction model based on the proposed MRG decoder. Firstly, we employ a Transformer as an encoder to extract encoded features from the past trajectory. Secondly, we introduce an interaction-aware latent code generator to learn a Gaussian distribution from the social context among pedestrians for latent code sampling. Finally, we feed the encoded features to the MRG decoder and sample the latent code multiple times from the learned Gaussian distribution, providing additional inputs to the MRG decoder to generate multiple socially acceptable future trajectories. Experimental results on two public datasets, ETH and UCY, validate the effectiveness of the MRGTraj model. Besides, the MRGTraj model achieves superior prediction performance, with improvements of 13.21% on FDE metrics and a 71.29% speed-up compared to state-of-the-art models. The code is available athttps://github.com/wisionpeng/MRGTraj.

Transformer has been widely used in histopathology whole slide image analysis. However, the design of token-wise self-attention and positional embedding strategy in the common Transformer limits its effectiveness and efficiency when applied to gigapixel histopathology images. In this paper, we propose a novel kernel attention Transformer (KAT) for histopathology WSI analysis and assistant cancer diagnosis. The information transmission in KAT is achieved by cross-attention between the patch features and a set of kernels related to the spatial relationship of the patches on the whole slide images. Compared to the common Transformer structure, KAT can extract the hierarchical context information of the local regions of the WSI and provide diversified diagnosis information. Meanwhile, the kernel-based cross-attention paradigm significantly reduces the computational amount. The proposed method was evaluated on three large-scale datasets and was compared with 8 state-of-the-art methods. The experimental results have demonstrated the proposed KAT is effective and efficient in the task of histopathology WSI analysis and is superior to the state-of-the-art methods.

Computer-aided cancer diagnosis (CAD) methods based on the histopathological images have achieved great development. The content-based whole slide image (WSI) retrieval is one of the important application that can search for the informative data to assist clinical diagnosis. It is notable that the current retrieval system are mainly developed based on the image content and image labels. The diagnosis report for the WSIs given by the pathologists are also valuable data, but have not yet been adequately considered in modeling. In this paper, we propose a cross-modal retrieval framework based on histopathology WSIs and diagnosis report, which can simultaneously achieve four retrieval tasks for histopathology database across WSIs and diagnosis reports. The compact binary features from both WSIs and diagnosis reports are first extracted, and then built in a common vision-language semantic feature space by the constraint of the designed cross hashing loss function. The method was verified on a gastric histopathology dataset that contains 932 gastric cases with 4 lesion categories. Experimental results have demonstrated the effectiveness of the proposed method in the cross-modal retrieval tasks for digital pathology system.

Color consistency is crucial to developing robust deep learning methods for histopathological image analysis. With the increasing application of digital histopathological slides, the deep learning methods are probably developed based on the data from multiple medical centers. This requirement makes it a challenging task to normalize the color variance of histopathological images from different medical centers. In this paper, we propose a novel color standardization module named stain standardization capsule based on the capsule network and the corresponding dynamic routing algorithm. The proposed module can learn and generate uniform stain separation outputs for histopathological images in various color appearance without the reference to manually selected template images. The proposed module is light and can be jointly trained with the application-driven CNN model. The proposed method was validated on three histopathology datasets and a cytology dataset, and was compared with state-of-the-art methods. The experimental results have demonstrated that the SSC module is effective in improving the performance of histopathological image analysis and has achieved the best performance in the compared methods.

The development of whole slide imaging techniques and online digital pathology platforms have accelerated the popularization of telepathology for remote tumor diagnoses. During a diagnosis, the behavior information of the pathologist can be recorded by the platform and then archived with the digital case. The browsing path of the pathologist on the WSI is one of the valuable information in the digital database because the image content within the path is expected to be highly correlated with the diagnosis report of the pathologist. In this article, we proposed a novel approach for computer-assisted cancer diagnosis named session-based histopathology image recommendation (SHIR) based on the browsing paths on WSIs. To achieve the SHIR, we developed a novel diagnostic regions attention network (DRA-Net) to learn the pathology knowledge from the image content associated with the browsing paths. The DRA-Net does not rely on the pixel-level or region-level annotations of pathologists. All the data for training can be automatically collected by the digital pathology platform without interrupting the pathologists' diagnoses. The proposed approaches were evaluated on a gastric dataset containing 983 cases within 5 categories of gastric lesions. The quantitative and qualitative assessments on the dataset have demonstrated the proposed SHIR framework with the novel DRA-Net is effective in recommending diagnostically relevant cases for auxiliary diagnosis. The MRR and MAP for the recommendation are respectively 0.816 and 0.836 on the gastric dataset. The source code of the DRA-Net is available at https://github.com/zhengyushan/dpathnet.

Land cover classification can be regarded as topic assignment that the pixels can be classified into different kinds of regions (e.g. road, tree, grass) according to the semantics of topics in topic model. In this paper, we present a novel probabilistic latent semantic analysis (pLSA) model based on sparsity constraint for classifying different kinds of land cover. In contrast with conventional topic model which usually assumes each local feature descriptor is only related to one visual word of the dictionary, our method uses sparse coding to characterize the potential relationship between the descriptor and multiple words. Therefore each descriptor can be represented by a small set of words. More importantly, we further apply sparse coding to mine the correlation of documents (i.e. image) in pLSA model. Consequently, our model can generate the more discriminative latent topics and benefit land cover classification. Experimental results on high-resolution remote sensing images demonstrate the excellent superiority of our method.

Content-based image retrieval (CBIR) for digital pathology slides is of clinical use for breast cancer aided diagnosis. One of the largest challenges in CBIR is feature extraction. In this paper, we propose a novel pathology image retrieval method for breast cancer, which aims to characterize the pathology image content through texture and pathological features and further discover the latent high-level semantics. Specifically, the proposed method utilizes block Gabor features to describe the texture structure, and simultaneously designs nucleus-based pathological features to describe morphological characteristics of nuclei. Based on these two kinds of local feature descriptors, two codebooks are built to learn the probabilistic latent semantic analysis (pLSA) models. Consequently, each image is represented by the topics of pLSA models which can reveal the semantic concepts. Experimental results on the digital pathology image database for breast cancer demonstrate the feasibility and effectiveness of our method.

The recent emerging sparse coding (SC) algorithms do not take local manifold structure of samples into consideration, while graph regularized sparse coding (GraphSC) algorithm only constrains the locality consistency of samples. Furthermore, the graph construction approach based on k-nearest-neighbor usually pre-defines the number of neighbors for all the samples, which may fails to fit the intrinsic structure of each sample. To address these issues, we propose an local and nonlocal graph regularized sparse coding (LN-GraphSC) algorithm. LN-GraphSC incorporates both local and nonlocal information of samples at the same time. On the other hand, to alleviate the problem of neighbor parameter selection, we use average distance of each sample to wisely determine its own local and nonlocal samples. To verify the effectiveness of our proposed method, we evaluate our method on the task of face recognition. The experimental results on ORL and Yale face databases show our method has competitive performance when compared to SC and GraphSC.

Pathological image retrieval contributes to computer-aided diagnosis for breast cancer due to the fact that the retrieval results generally contain detailed diagnostic information (e.g. abnormal regions and diagnostic opinion from other doctors) which can offer some reference and assistance to the doctor during diagnosis process. In this paper, we present a novel pathological image retrieval approach based on probabilistic latent semantic analysis (pLSA) model. The method respectively utilizes SIFT features after visual saliency detection, and block Gabor features for the construction of two semantic codebooks, which not only can characterize the salient local invariant features and texture information under different scales and orientations in the pathological images, but also consider the high-level semantic features. Furthermore, we apply pLSA model to discover the latent topics in each codebook. Finally each pathological image is represented by the combination of topics from these two codebooks. The proposed method is evaluated on the pathological image database for breast cancer, which includes 5 categories (mucinous cystadenocarcinoma, invasive lobular carcinoma, basal-like carcinoma, invasive breast cancer and low-grade adenosquamous carcinoma) and 110 cases for each category. Experimental results demonstrate the feasibility and effectiveness of our method.

Land cover segmentation can be viewed as topic assignment that the pixels are grouped into homogeneous regions according to different semantic topics in topic model. In this paper, we propose a novel topic model based on sparse coding for segmenting different kinds of land covers. Different from conventional topic models which generally assume each local feature descriptor is related to only one visual word of the codebook, our method utilizes sparse coding to characterize the potential correlation between the descriptor and multiple words. Therefore each descriptor can be represented by a small set of words. Furthermore, in this paper probabilistic Latent Semantic Analysis (pLSA) is applied to learn the latent relation among word, topic and document due to its simplicity and low computational cost. Experimental results on remote sensing image segmentation demonstrate the excellent superiority of our method over k-means clustering and conventional pLSA model.

Bag-of-features (BoF) model based on SIFT generally assumes each descriptor is related to only one visual word of the codebook. Therefore, the potential correlation between the descriptor and other visual words is ignored. On the other hand, sparse coding through l1-norm regularization fails to generate optimal sparse representations since l1-norm regularization randomly selected one variable from a group of highly correlated variables. In this study we propose a novel bag-of-features model for image retrieval called SIFT-based Elastic sparse coding. The method utilizes a large number of SIFT descriptors to construct the codebook. The Elastic Net regression framework, which combines both l1-norm and l2-norm penalties, is then used to obtain the sparse-coefficient vector corresponding to the SIFT descriptor. Finally each image can be represented by a unified sparse-coefficient vector. Experimental results on Coil20 dataset demonstrate the consistent superiority of the proposed method over the state-of-the-art algorithms including original SIFT matching, conventional BoF strategy and BoF model based on l1-norm sparse coding.