Multiple instance learning (MIL) has emerged as a prominent paradigm for processing the whole slide image with pyramid structure and giga-pixel size in digital pathology. However, existing attention-based MIL methods are primarily trained on the image modality and a pre-defined label set, leading to limited generalization and interpretability. Recently, vision language models (VLM) have achieved promising performance and transferability, offering potential solutions to the limitations of MIL-based methods. Pathological diagnosis is an intricate process that requires pathologists to examine the WSI step-by-step. In the field of natural language process, the chain-of-thought (CoT) prompting method is widely utilized to imitate the human reasoning process. Inspired by the CoT prompt and pathologists' clinic knowledge, we propose a chain-of-diagnosis prompting multiple instance learning (CoD-MIL) framework for whole slide image classification. Specifically, the chain-of-diagnosis text prompt decomposes the complex diagnostic process in WSI into progressive sub-processes from low to high magnification. Additionally, we propose a text-guided contrastive masking module to accurately localize the tumor region by masking the most discriminative instances and introducing the guidance of normal tissue texts in a contrastive way. Extensive experiments conducted on three real-world subtyping datasets demonstrate the effectiveness and superiority of CoD-MIL.

Accurate tissue segmentation in histopathological images is essential for promoting the development of precision pathology. However, the size of the digital pathological image is great, which needs to be tiled into small patches containing limited semantic information. To imitate the pathologist's diagnosis process and model the semantic relation of the whole slide image, We propose a semi-supervised pixel contrastive learning framework (SSPCL) which mainly includes an uncertainty-guided mutual dual consistency learning module (UMDC) and a cross image pixel-contrastive learning module (CIPC). The UMDC module enables efficient learning from unlabeled data through mutual dual-consistency and consensus-based uncertainty. The CIPC module aims at capturing the cross-patch semantic relationship by optimizing a contrastive loss between pixel embeddings. We also propose several novel domain-related sampling methods by utilizing the continuous spatial structure of adjacent image patches, which can avoid the problem of false sampling and improve the training efficiency. In this way, SSPCL significantly reduces the labeling cost on histopathological images and realizes the accurate quantitation of tissues. Extensive experiments on three tissue segmentation datasets demonstrate the effectiveness of SSPCL, which outperforms state-of-the-art up to 5.0% in mDice.

Multiple instance learning (MIL)-based methods have become the mainstream for processing the megapixel-sized whole slide image (WSI) with pyramid structure in the field of digital pathology. The current MIL-based methods usually crop a large number of patches from WSI at the highest magnification, resulting in a lot of redundancy in the input and feature space. Moreover, the spatial relations between patches can not be sufficiently modeled, which may weaken the model's discriminative ability on fine-grained features. To solve the above limitations, we propose a Multi-scale Graph Transformer (MG-Trans) with information bottleneck for whole slide image classification. MG-Trans is composed of three modules: patch anchoring module (PAM), dynamic structure information learning module (SILM), and multi-scale information bottleneck module (MIBM). Specifically, PAM utilizes the class attention map generated from the multi-head self-attention of vision Transformer to identify and sample the informative patches. SILM explicitly introduces the local tissue structure information into the Transformer block to sufficiently model the spatial relations between patches. MIBM effectively fuses the multi-scale patch features by utilizing the principle of information bottleneck to generate a robust and compact bag-level representation. Besides, we also propose a semantic consistency loss to stabilize the training of the whole model. Extensive studies on three subtyping datasets and seven gene mutation detection datasets demonstrate the superiority of MG-Trans.

Pathological primary tumor (pT) stage focuses on the infiltration degree of the primary tumor to surrounding tissues, which relates to the prognosis and treatment choices. The pT staging relies on the field-of-views from multiple magnifications in the gigapixel images, which makes pixel-level annotation difficult. Therefore, this task is usually formulated as a weakly supervised whole slide image (WSI) classification task with the slide-level label. Existing weakly-supervised classification methods mainly follow the multiple instance learning paradigm, which takes the patches from single magnification as the instances and extracts their morphological features independently. However, they cannot progressively represent the contextual information from multiple magnifications, which is critical for pT staging. Therefore, we propose a structure-aware hierarchical graph-based multi-instance learning framework (SGMF) inspired by the diagnostic process of pathologists. Specifically, a novel graph-based instance organization method is proposed, namely structure-aware hierarchical graph (SAHG), to represent the WSI. Based on that, we design a novel hierarchical attention-based graph representation (HAGR) network to capture the critical patterns for pT staging by learning cross-scale spatial features. Finally, the top nodes of SAHG are aggregated by a global attention layer for bag-level representation. Extensive studies on three large-scale multi-center pT staging datasets with two different cancer types demonstrate the effectiveness of SGMF, which outperforms state-of-the-art up to 5.6% in the F1 score.

Computational Pathology (CPATH) offers the possibility for highly accurate and low-cost automated pathological diagnosis. However, the high time cost of model inference is one of the main issues limiting the application of CPATH methods. Due to the large size of Whole-Slide Image (WSI), commonly used CPATH methods divided a WSI into a large number of image patches at relatively high magnification, then predicted each image patch individually, which is time-consuming. In this paper, we propose a novel Uncertainty-based Model Acceleration (UMA) method for reducing the time cost of model inference, thereby relieving the deployment burden of CPATH applications. Enlightened by the slide-viewing process of pathologists, only a few high-uncertain regions are regarded as “suspicious” regions that need to be predicted at high magnification, and most of the regions in WSI are predicted at low magnification, thereby reducing the times of image patch extraction and prediction. Meanwhile, uncertainty estimation ensures prediction accuracy at low magnification. We take two fundamental CPATH classification tasks (i.e., cancer region detection and subtyping) as examples. Extensive experiments on two large-scale renal cell carcinoma classification datasets demonstrate that our UMA can significantly reduce the time cost of model inference while maintaining competitive classification performance.

Diagnostic pathology is the foundation and gold standard for identifying carcinomas, and the accurate quantification of pathological images can provide objective clues for pathologists to make more convincing diagnosis. Recently, the encoder-decoder architectures (EDAs) of convolutional neural networks (CNNs) are widely used in the analysis of pathological images. Despite the rapid innovation of EDAs, we have conducted extensive experiments based on a variety of commonly used EDAs, and found them cannot handle the interference of complex background in pathological images, making the architectures unable to focus on the regions of interest (RoIs), thus making the quantitative results unreliable. Therefore, we proposed a pathway named GLobal Bank (GLB) to guide the encoder and the decoder to extract more features of RoIs rather than the complex background. Sufficient experiments have proved that the architecture remoulded by GLB can achieve significant performance improvement, and the quantitative results are more accurate.

Tissue segmentation is an essential task in computational pathology. However, relevant datasets for such a pixel-level classification task are hard to obtain due to the difficulty of annotation, bringing obstacles for training a deep learning-based segmentation model. Recently, contrastive learning has provided a feasible solution for mitigating the heavy reliance of deep learning models on annotation. Nevertheless, applying contrastive loss to the most abstract image representations, existing contrastive learning frameworks focus on global features, therefore, are less capable of encoding finer-grained features (e.g., pixel-level discrimination) for the tissue segmentation task. Enlightened by domain knowledge, we design three contrastive learning tasks with multi-granularity views (from global to local) for encoding necessary features into representations without accessing annotations. Specifically, we construct: (1) an image-level task to capture the difference between tissue components, i.e., encoding the component discrimination; (2) a superpixel-level task to learn discriminative representations of local regions with different tissue components, i.e., encoding the prototype discrimination; (3) a pixel-level task to encourage similar representations of different tissue components within a local region, i.e., encoding the spatial smoothness. Through our global-to-local pre-training strategy, the learned representations can reasonably capture the domain-specific and fine-grained patterns, making them easily transferable to various tissue segmentation tasks in histopathological images. We conduct extensive experiments on two tissue segmentation datasets, while considering two real-world scenarios with limited or sparse annotations. The experimental results demonstrate that our framework is superior to existing contrastive learning methods and can be easily combined with weakly supervised and semi-supervised segmentation methods.

Nuclei segmentation is a fundamental task in digital pathology analysis and can be automated by deep learning-based methods. However, the development of such an automated method requires a large amount of data with precisely annotated masks which is hard to obtain. Training with weakly labeled data is a popular solution for reducing the workload of annotation. In this paper, we propose a novel meta-learning-based nuclei segmentation method which follows the label correction paradigm to leverage data with noisy masks. Specifically, we design a fully conventional meta-model that can correct noisy masks using a small amount of clean meta-data. Then the corrected masks can be used to supervise the training of the segmentation model. Meanwhile, a bi-level optimization method is adopted to alternately update the parameters of the main segmentation model and the meta-model in an end-to-end way. Extensive experimental results on two nuclear segmentation datasets show that our method achieves the state-of-the-art result. It even achieves comparable performance with the model training on supervised data in some noisy settings.

Digital pathology plays a crucial role in the development of artificial intelligence in the medical field. The digital pathology platform can make the pathological resources digital and networked, and realize the permanent storage of visual data and the synchronous browsing processing without the limitation of time and space. It has been widely used in various fields of pathology. However, there is still a lack of an open and universal digital pathology platform to assist doctors in the management and analysis of digital pathological sections, as well as the management and structured description of relevant patient information. Most platforms cannot integrate image viewing, annotation and analysis, and text information management. To solve the above problems, we propose a comprehensive and extensible platform, PIMIP (Pathology Information Management & Integration Platform). PIMIP has developed the image annotation functions based on the visualization of digital pathological sections. Our annotation functions support multi-user collaborative annotation and multi-device annotation, and realize the automation of some annotation tasks. In the annotation task, we invited a professional pathologist for guidance. We introduce a machine learning module for image analysis. The data we collected included public data from local hospitals and clinical examples. Our platform is more clinical and suitable for clinical use. In addition to image data, we also structured the management and display of text information. So our platform is comprehensive. The platform framework is built in a modular way to support users to add machine learning modules independently, which makes our platform extensible.

Diagnostic pathology, which is the basis and gold standard of cancer diagnosis, provides essential information on the prognosis of the disease and vital evidence for clinical treatment. However, pathological diagnosis is subjective, and differences in observation and diagnosis between pathologists are common. This phenomenon is more evident in hospitals with insufficient medical resources. Deep learning (DL) can be used to identify and classify structures in digital pathology. In order to solve the above difficulties, in this work, we propose a DL framework for generating pathological diagnosis by analyzing histopathological images of renal cell carcinoma. A deep neural network is trained on a large high-quality annotated dataset for accurate tumor area detection, subtyping, and grading. The results show that our framework has achieved pathologist-level accuracy in diagnosis, can generate pathology reports with tumor indicators, and provide pathologists with interpretable auxiliary diagnoses

Constructing large-scaled medical knowledge graphs (MKGs) can significantly boost healthcare applications for medical surveillance, bring much attention from recent research. An essential step in constructing large-scale MKG is extracting information from medical reports. Recently, information extraction techniques have been proposed and show promising performance in biomedical information extraction. However, these methods only consider limited types of entity and relation due to the noisy biomedical text data with complex entity correlations. Thus, they fail to provide enough information for constructing MKGs and restrict the downstream applications. To address this issue, we propose Biomedical Information Extraction (BioIE), a hybrid neural network to extract relations from biomedical text and unstructured medical reports. Our model utilizes a multi-head attention enhanced graph convolutional network (GCN) to capture the complex relations and context information while resisting the noise from the data. We evaluate our model on two major biomedical relationship extraction tasks, chemical-disease relation (CDR) and chemical-protein interaction (CPI), and a cross-hospital pan-cancer pathology report corpus. The results show that our method achieves superior performance than baselines. Furthermore, we evaluate the applicability of our method under a transfer learning setting and show that BioIE achieves promising performance in processing medical text from different formats and writing styles.

Personalized diagnoses have not been possible due to a sear amount of data pathologists have to bear during the day-to-day routine, leading to the current generalized standards being continuously updated as new findings are reported. It is noticeable that these practical standards are developed based on multi-source heterogeneous data, including whole-slide images and pathology and clinical reports. In this study, we propose a framework that combines pathological images and medical reports to generate a personalized diagnosis result for an individual patient. We use nuclei-level image feature similarity and content-based deep learning method to search for a personalized group of populations with similar pathological characteristics, extract structured prognostic information from descriptive pathology reports of the similar patient population, and assign importance of different prognostic factors to generate a personalized pathological diagnosis result. We use multi-source heterogeneous data from TCGA (The Cancer Genome Atlas) database. The result demonstrates that our framework matches the performance of pathologists in the diagnosis of renal cell carcinoma. This framework is designed to be generic, and this could be applied to other types of cancer. The weights could provide insights into the known prognostic factors and further guide more precise clinical treatment protocols.

Electronic medical data contains biochemical, imaging, pathological information during diagnosis and treatment. The pathology report is a kind of highly liberalized unstructured textual data, which is the basis and gold standard of cancer diagnosis and is very important for the prognosis and treatment of patients. The application of information extraction technology to pathological reports can obtain structured data that can be understood and analyzed by computers, helping pathologists make appropriate decisions. In this work, we proposed an attention-based graph convolutional network (GCN) for converting unstructured pathological reports into a structured form suitable for computer analysis to improve the current pathologist's workflow, collected medical data from different platforms, and provided more accurate assistance for diagnosis and treatment. We used pathology reports data from TCGA (The Cancer Genome Atlas) database with fine-grained annotations on 3632 pathology reports including four types of cancers. Our method performs better in our pathology report dataset with higher F1 score than traditional methods and deep learning methods. The results indicate that our method is robust, thus may work with other types of cancer pathology report.

The encoder-decoder architecture of convolutional neural networks (CNNs) is widely used in computer vision tasks and various analyses of medical images. However, extracting semantic features from regions of interest (RoIs) in pathological images remains a challenging task because RoIs of different morphologies and scales are embedded in a blurred background. Additionally, it is well known that the classic encoder-decoder architecture is vulnerable to interference from a blurred background and is thus not entirely suitable for precise analysis of pathological images. In this paper, we propose a pathway named global bank (GLB) to guide the encoder and decoder to focus more on the RoIs by providing the decoder with additional effective features of the RoIs. We extend the U-Net and feature pyramid network (FPN) with GLB and evaluate the resulting models on gland segmentation and cancer embolus detection tasks, respectively. Extensive experiments demonstrate that our proposal can significantly improve the performance of the encoder-decoder architecture. The U-Net with GLB achieves the best semantic segmentation performance on the 2015 MICCAI Gland Challenge dataset. Additionally, the FPN with GLB achieves improvements of 2% in average precision and 3.4% in recall on the embolus detection task.

Transition from conventional to digital pathology requires a new category of biomedical informatic infrastructure which could facilitate delicate pathological routine. Pathological diagnoses are sensitive to many external factors and is known to be subjective. Only systems that can meet strict requirements in pathology would be able to run along pathological routines and eventually digitized the area, and the developed platform should comply with existing pathological routines and international standards. Currently, there are a number of available software tools which can perform histopathological tasks including virtual slide viewing, annotating, and basic image analysis, however, none of them can serve as a digital platform for pathology. Here we describe OpenHI2, an enhanced version Open Histopathological Image platform which is capable of supporting all basic pathological tasks and file formats; ready to be deployed in medical institutions on a standard server environment or cloud computing infrastructure. In this paper, we also describe the development decisions for the platform and propose solutions to overcome technical challenges including responsive region retrieval and viewing, virtual slide magnification, recording of diagnostic areas. These factors would promote OpenHI2 be used as a platform for histopathological images in real-world clinical settings. Furthermore, in research, OpenHI2 inherited the annotation functionality from the previous version, thus acquired annotations can be directly utilized by the newly added machine learning module which include popular machine learning models to perform tasks such as histology image classification and segmentation in the same environment. Addition can be made to the platform since each component is modularized and fully documented. OpenHI2 is free, open-source, and available at https://gitlab.com/BioAI/OpenHI.

Pathological is crucial to cancer diagnosis. Usually, Pathologists draw their conclusion based on observed cell and tissue structure on histology slides. Rapid development in machine learning, especially deep learning have established robust and accurate classifiers. They are being used to analyze histopathological slides and assist pathologists in diagnosis. Most machine learning systems rely heavily on annotated data sets to gain experiences and knowledge to correctly and accurately perform various tasks such as classification and segmentation. Generally, annotations made in pathology-related datasets have inherited annotation methods from natural scene images. This work investigates different granularity of annotations in histopathological data set including image-wise, bounding box, ellipse-wise, and pixel-wise to verify the influence of annotation in pathological slide on deep learning models. We design corresponding experiments to test classification and segmentation performance of deep learning models based on annotations with different annotation granularity. In classification, state-of-the-art deep learning-based classifiers perform better when trained by pixel-wise annotation dataset. On average, precision, recall and F1-score improves by 7.87%, 8.83% and 7.85% respectively. Thus, it is suggested that finer granularity annotations are better utilized by deep learning algorithms in classification tasks. Similarly, semantic segmentation algorithms can achieve 8.33% better segmentation accuracy when trained by pixel-wise annotations. Our study shows not only that finer-grained annotation can improve the performance of deep learning models, but also help they extract more accurate phenotypic information from histopathological slides. The accurate and spatially precise acquisitions of phenotypic information can improve the reliability of the model prediction. Intelligence systems trained on granular annotations may help pathologists inspecting certain regions and features in the slide that were mainly used to calculate the prediction. The compartmentalized prediction approach similar to this work may contribute to phenotype and genotype association studies.

Information in histology slides are usually visualized by different staining techniques, each of them unveils specific chemical and biological substances within tissue samples. Correlations between different stains can be useful to predict how certain tissue slides may look like if they were stained by other staining techniques. This work investigates two stains including hematoxylin and eosin (H&E) and immunohistochemistry (IHC) in digital pathological slides. Four cases of surgical biopsies were used in this work. The specimens were subjected to two consecutive stains with a decoloring process based on ethanol and potassium permanganate in between. After each stain, slides were digitized and archived as results. Comparing the effects of the two staining pipelines, IHC slides after decoloring of H&E showed that the cell structure was clear, the positive IHC staining was accurate, the background of the slide was clean, there was no DAB residue, and tissue fragments were intact. However, the other pipeline where IHC was stained before H&E showed that the nuclear border was blurred. Eosin is lightly colored resulting in low contrast visualization of nucleoplasm, DAB is not completely decolored, and parts of tissue were fragmented. We conclude that, from the proposed staining and decoloring technique, tissue slides could be stained with IHC more effectively on decolored H&E slides than those stained with H&E after IHC. Utilizing digital section scanning technology, we can obtain pairs of tissue images stained differently while preserving the exact same tissue structure.