In the field of medical image analysis, medical image classification is one of the most fundamental and critical tasks. Current researches often rely on the off-the-shelf backbone networks derived from the field of computer vision, hoping to achieve satisfactory classification performance for medical images. However, given the characteristics of medical images, such as scattered distribution and varying sizes of lesions, features extracted with a single scale from the existing backbones often fail to perform accurate medical image classification. To this end, we propose a novel multi-scale learning paradigm, namely MUlti-SCale Learning with trusted Evidences (MUSCLE), which extracts and integrates features from different scales based on shape the theory of evidence, to generate the more comprehensive feature representation for the medical image classification task. Particularly, the proposed MUSCLE first estimates the uncertainties of features extracted from different scales/stages of the classification backbone as the evidences, and accordingly form the opinions regarding to the feature trustworthiness via a set of evidential deep neural networks. Then, these opinions on different scales of features are ensembled to yield an aggregated opinion, which can be used to adaptively tune the weights of multi-scale features for scatteredly distributed and size-varying lesions, and consequently improve the network capacity for accurate medical image classification. Our MUSCLE paradigm has been evaluated on five publicly available medical image datasets. The experimental results show that the proposed MUSCLE not only improves the accuracy of the original backbone network, but also enhances the reliability and interpretability of model decisions with the trusted evidences (https://github.com/Q4CS/MUSCLE).

Immunohistochemical (IHC) biomarker prediction greatly benefits from multimodal data fusion. However, the simultaneous acquisition of genomic and pathological data is often constrained by cost or technical limitations. To address this, we propose a novel Genomics-guided Multimodal Knowledge Decomposition Network (GMKDN), a framework that effectively integrates genomics and pathology data during training while dynamically adapting to available data during inference. GMKDN introduces two key innovations: 1) the Batch-Sample Multimodal Knowledge Decomposition (BMKD) module, which decomposes input features into pathology-specific, modality-general, and genomics-specific components to reduce redundancy and enhance knowledge transferability, and 2) the Online Similarity-Preserving Knowledge Distillation (OSKD) module, which optimizes activation similarity matrices to facilitate robust knowledge transfer between teacher and student models. The BMKD module improves generalization across modalities, while the OSKD module enhances model robustness, particularly when certain modalities are unavailable during inference. Extensive evaluations conducted on the TCGA-BRCA dataset and an external test cohort (QHSU) demonstrate that GMKDN consistently outperforms state-of-the-art (SOTA) slide-based multiple instance learning (MIL) approaches as well as existing multimodal learning models, establishing a new benchmark for breast cancer biomarker prediction. Our code is available at https://github.com/qiyuanzz/GMKDN.

To improve the overall survival rate of cancer patients, we propose an innovative approach named Multimodal Fusion Framework based on Low-rank Interaction (MF2LI), which aims to overcome the current limitations of relying solely on single-modal data prediction and the excessive complexity of fusion. By harnessing low-rank multimodal fusion (LMF) and optimal weight integration (OWI), MF2LI maximizes the integration of pathological images and genomic data. The model incorporates a parallel decomposition strategy, reducing complexity and facilitating fusion based on the contributions of each component. We validate our method using the GBMLGG and KIRC datasets from The Cancer Genome Atlas (TCGA). The C-index of the proposed model stands at $0.895 \pm 0.007$ and $0.728 \pm 0.030$ for the two datasets, respectively, outperforming existing methods. Furthermore, we generate visualizations of the risk ratios, which demonstrate a strong alignment with the actual grade classifications. Extensive experiments have shown that our model improves the prognosis prediction of tumor patients and has considerable clinical value.

To accurately evaluate the patient’s condition, medical workers usually need to register multiple pathological images of the lesion site samples. Using computer technology to assist in registration work can effectively improve the efficiency of doctors analyzing pathological images. One of the most advanced methods currently is the Virtual Alignment of Pathology Image Series method, which is a multi-staining digital pathology image registration method that combines global and local calculations. However, this method may encounter certain biases when processing images with significant angle differences. Through a detailed analysis of this method, this article proposes an improvement plan which optimizes the acquisition of non-rigid registration mask images, enabling the method to obtain mask images more reasonably and achieve better registration results for images with significant angle differences. This provides more accurate judgment basis and helps doctors diagnose and develop treatment plans more accurately.

Predicting the gene mutation status in whole slide images (WSIs) is crucial for the clinical treatment, cancer management, and research of gliomas. With advancements in CNN and Transformer algorithms, several promising models have been proposed. However, existing studies have paid little attention on fusing multi-magnification information, and the model requires processing all patches from a whole slide image. In this paper, we propose a cross-magnification attention model called CroMAM for predicting the genetic status and survival of gliomas. The CroMAM first utilizes a systematic patch extraction module to sample a subset of representative patches for downstream analysis. Next, the CroMAM applies Swin Transformer to extract local and global features from patches at different magnifications, followed by acquiring high-level features and dependencies among single-magnification patches through the application of a Vision Transformer. Subsequently, the CroMAM exchanges the integrated feature representations of different magnifications and encourage the integrated feature representations to learn the discriminative information from other magnification. Additionally, we design a cross-magnification attention analysis method to examine the effect of cross-magnification attention quantitatively and qualitatively which increases the model's explainability. To validate the performance of the model, we compare the proposed model with other multi-magnification feature fusion models on three tasks in two datasets. Extensive experiments demonstrate that the proposed model achieves state-of-the-art performance in predicting the genetic status and survival of gliomas.

Magnetic resonance imaging (MRI)-based deep neural networks (DNN) have been widely developed to perform prostate cancer (PCa) classification. However, in real-world clinical situations, prostate MRIs can be easily impacted by rectal artifacts, which have been found to lead to incorrect PCa classification. Existing DNN-based methods typically do not consider the interference of rectal artifacts on PCa classification, and do not design specific strategy to address this problem. In this study, we proposed a novel Targeted adversarial training with Proprietary Adversarial Samples (TPAS) strategy to defend the PCa classification model against the influence of rectal artifacts. Specifically, based on clinical prior knowledge, we generated proprietary adversarial samples with rectal artifact-pattern adversarial noise, which can severely mislead PCa classification models optimized by the ordinary training strategy. We then jointly exploited the generated proprietary adversarial samples and original samples to train the models. To demonstrate the effectiveness of our strategy, we conducted analytical experiments on multiple PCa classification models. Compared with ordinary training strategy, TPAS can effectively improve the single- and multi-parametric PCa classification at patient, slice and lesion level, and bring substantial gains to recent advanced models. In conclusion, TPAS strategy can be identified as a valuable way to mitigate the influence of rectal artifacts on deep learning models for PCa classification.

Although deep learning (DL) has demonstrated impressive diagnostic performance for a variety of computational pathology tasks, this performance often markedly deteriorates on whole slide images (WSI) generated at external test sites. This phenomenon is due in part to domain shift, wherein differences in test-site pre-analytical variables (e.g., slide scanner, staining procedure) result in WSI with notably different visual presentations compared to training data. To ameliorate pre-analytic variances, approaches such as CycleGAN can be used to calibrate visual properties of images between sites, with the intent of improving DL classifier generalizability. In this work, we present a new approach termed Multi-Site Cross-Organ Calibration based Deep Learning (MuSClD) that employs WSIs of an off-target organ for calibration created at the same site as the on-target organ, based off the assumption that cross-organ slides are subjected to a common set of pre-analytical sources of variance. We demonstrate that by using an off-target organ from the test site to calibrate training data, the domain shift between training and testing data can be mitigated. Importantly, this strategy uniquely guards against potential data leakage introduced during calibration, wherein information only available in the testing data is imparted on the training data. We evaluate MuSClD in the context of the automated diagnosis of non-melanoma skin cancer (NMSC). Specifically, we evaluated MuSClD for identifying and distinguishing (a) basal cell carcinoma (BCC), (b) in-situ squamous cell carcinomas (SCC-In Situ), and (c) invasive squamous cell carcinomas (SCC-Invasive), using an Australian (training, n = 85) and a Swiss (held-out testing, n = 352) cohort. Our experiments reveal that MuSCID reduces the Wasserstein distances between sites in terms of color, contrast, and brightness metrics, without imparting noticeable artifacts to training data. The NMSC-subtyping performance is statistically improved as a result of MuSCID in terms of one-vs. rest AUC: BCC (0.92 vs 0.87, p = 0.01), SCC-In Situ (0.87 vs 0.73, p = 0.15) and SCC-Invasive (0.92 vs 0.82, p = 1e-5). Compared to baseline NMSC-subtyping with no calibration, the internal validation results of MuSClD (BCC (0.98), SCC-In Situ (0.92), and SCC-Invasive (0.97)) suggest that while domain shift indeed degrades classification performance, our on-target calibration using off-target tissue can safely compensate for pre-analytical variabilities, while improving the robustness of the model.

Brain tumor segmentation (BTS) in magnetic resonance image (MRI) is crucial for brain tumor diagnosis, cancer management and research purposes. With the great success of the ten-year BraTS challenges as well as the advances of CNN and Transformer algorithms, a lot of outstanding BTS models have been proposed to tackle the difficulties of BTS in different technical aspects. However, existing studies hardly consider how to fuse the multi-modality images in a reasonable manner. In this paper, we leverage the clinical knowledge of how radiologists diagnose brain tumors from multiple MRI modalities and propose a clinical knowledge-driven brain tumor segmentation model, called CKD-TransBTS. Instead of directly concatenating all the modalities, we re-organize the input modalities by separating them into two groups according to the imaging principle of MRI. A dual-branch hybrid encoder with the proposed modality-correlated cross-attention block (MCCA) is designed to extract the multi-modality image features. The proposed model inherits the strengths from both Transformer and CNN with the local feature representation ability for precise lesion boundaries and long-range feature extraction for 3D volumetric images. To bridge the gap between Transformer and CNN features, we propose a Trans&CNN Feature Calibration block (TCFC) in the decoder. We compare the proposed model with six CNN-based models and six transformer-based models on the BraTS 2021 challenge dataset. Extensive experiments demonstrate that the proposed model achieves state-of-the-art brain tumor segmentation performance compared with all the competitors.

Numerous microbes have been found to have vital impacts on human health through affecting biological processes. Therefore, exploring potential associations between microbes and diseases will promote the understanding and diagnosis of diseases. In this study, we present a novel computational model, named MSLINE, to infer potential microbe-disease associations by integrating Multiple Similarities and Large-scale Information Network Embedding (LINE) based on known associations. Specifically, on the basis of known microbe-disease associations from the Human Microbe-Disease Association Database, we first increase the known associations by collecting proven associations from existing literatures. We then construct a microbe-disease heterogeneous network (MDHN) by integrating known associations and multiple similarities (including Gaussian interaction profile kernel similarity, microbe function similarity, disease semantic similarity and disease-symptom similarity). After that, we implement random walk and LINE algorithm on MDHN to learn its structure information. Finally, we score the microbe-disease associations according to the structure information for every nodes. In the Leave-one-out cross validation and 5-fold cross validation, MSLINE performs better compared to other existing methods. Moreover, case studies of different diseases proved that MSLINE could predict the potential microbe-disease associations efficiently.

Almost 25% of COVID-19 patients end up in ICU needing critical mechanical ventilation support. There is currently no validated objective way to predict which patients will end up needing ventilator support, when the disease is mild and not progressed. N = 869 patients from two sites (D1: N = 822, D2: N = 47) with baseline clinical characteristics and chest CT scans were considered for this study. The entire dataset was randomly divided into 70% training, D1train(N = 606) and 30% test-set (Dtest: D1test(N = 216) + D2(N = 47)). An expert radiologist delineated ground-glass-opacities (GGOs) and consolidation regions on a subset of D1train, (D1train_sub, N = 88). These regions were automatically segmented and used along with their corresponding CT volumes to train an imaging AI predictor (AIP) on D1trainto predict the need of mechanical ventilators for COVID-19 patients. Finally, top five prognostic clinical factors selected using univariate analysis were integrated with AIP to construct an integrated clinical and AI imaging nomogram (ClAIN). Univariate analysis identified lactate dehydrogenase, prothrombin time, aspartate aminotransferase, %lymphocytes, albumin as top five prognostic clinical features. AIP yielded an AUC of 0.81 on Dtestand was independently prognostic irrespective of other clinical parameters on multivariable analysis (ptest. ClAIN outperformed AIP in predicting which COVID-19 patients ended up needing a ventilator. Our results across multiple sites suggest that ClAIN could help identify COVID-19 with severe disease more precisely and likely to end up on a life-saving mechanical ventilation.

Efficient and accurate detection of cell nuclei is an important step toward automatic analysis in histopathology. In this work, we present an automatic technique based on generalized Laplacian of Gaussian (gLoG) filter for nuclei detection in digitized histological images. The proposed technique first generates a bank of gLoG kernels with different scales and orientations and then performs convolution between directional gLoG kernels and the candidate image to obtain a set of response maps. The local maxima of response maps are detected and clustered into different groups by mean-shift algorithm based on their geometrical closeness. The point which has the maximum response in each group is finally selected as the nucleus seed. Experimental results on two datasets show that the proposed technique provides a superior performance in nuclei detection compared to existing techniques.

With the development of high‐speed, high‐resolution whole slide histology digital scanners, glass slides of tissue specimen can now be digitised at high magnification to create the whole slide image. Quantitative image analysis tools are then desirable to help the pathologist for their routine examination. Epidermis area is a very important observation area for the cancer diagnosis. Therefore, in order to build up a computer‐aided diagnosis system, segmentation of the epidermis area is often the very first and crucial step. An improved computer‐aided epidermis segmentation technique for the whole slide skin histopathological image is proposed in this study. The proposed technique first obtains an initial segmentation result with the help of global thresholding and shape analysis. A template matching method, with adaptive template intensity value, is then applied. Finally, a threshold is calculated based on the probability density function of the response value image. Experimental results show that the proposed technique overcomes the limitation of the existing technique and provides superior performance, with sensitivity of 95.68%, specificity of 99.41% and precision of 93.13%. The performance of the proposed technique is satisfactory for future clinical use.

The count of mitotic cells is a critical factor in most cancer grading systems. Extracting the mitotic cell from the histopathological image is a very challenging task. In this paper, we propose an efficient technique for detecting and segmenting the mitotic cells in the high-resolution multispectral image. The proposed technique consists of three main modules: discriminative image generation, mitotic cell candidate detection and segmentation, and mitotic cell candidate classification. In the first module, a discriminative image is obtained by linear discriminant analysis using ten different spectral band images. A set of mitotic cell candidate regions is then detected and segmented by the Bayesian modeling and local-region threshold method. In the third module, a 226 dimension feature is extracted from the mitotic cell candidates and their surrounding regions. An imbalanced classification framework is then applied to perform the classification for the mitotic cell candidates in order to detect the real mitotic cells. The proposed technique has been evaluated on a publicly available dataset of 35 × 10 multispectral images, in which 224 mitotic cells are manually labeled by experts. The proposed technique is able to provide superior performance compared to the existing technique, 81.5% sensitivity rate and 33.9% precision rate in terms of detection performance, and 89.3% sensitivity rate and 87.5% precision rate in terms of segmentation performance.

In this paper, we propose an efficient method for segmenting cell nuclei in the skin histopathological images. The proposed technique consists of four modules. First, it separates the nuclei regions from the background with an adaptive threshold technique. Next, an elliptical descriptor is used to detect the isolated nuclei with elliptical shapes. This descriptor classifies the nuclei regions based on two ellipticity parameters. Nuclei clumps and nuclei with irregular shapes are then localized by an improved seed detection technique based on voting in the eroded nuclei regions. Finally, undivided nuclei regions are segmented by a marked watershed algorithm. Experimental results on 114 different image patches indicate that the proposed technique provides a superior performance in nuclei detection and segmentation.

Detection of singular points (SPs) in fingerprint images is an important task in fingerprint recognition. In this paper, we propose a novel technique for SPs detection using orientation field regularization and the Poincaré Index (PI) technique. The squared orientation field is first extracted from a fingerprint image. In order to distinguish the local orientation patterns of genuine SPs from that of spurious SPs, a novel technique based on the Discrete Hodge Helmholtz Decomposition (DHHD) is proposed to reconstruct a regular orientation field of the fingerprint. Based on the regular orientation field, the PI technique is then applied to extract the SPs. Experimental results on the public fingerprint database FVC2002 show that, the proposed technique is rather accurate and robust in identifying SPs.

In the diagnosis of skin melanoma by analyzing histopathological images, the detection of the melanocytes in the epidermis area is an important step. However, the detection of melanocytes in the epidermis area is dicult because other keratinocytes that are very similar to the melanocytes are also present. This paper proposes a novel computer-aided technique for segmentation of the melanocytes in the skin histopathological images. In order to reduce the local intensity variant, a mean-shift algorithm is applied for the initial segmentation of the image. A local region recursive segmentation algorithm is then proposed to filter out the candidate nuclei regions based on the domain prior knowledge. To distinguish the melanocytes from other keratinocytes in the epidermis area, a novel descriptor, named local double ellipse descriptor (LDED), is proposed to measure the local features of the candidate regions. The LDED uses two parameters: region ellipticity and local pattern characteristics to distinguish the melanocytes from the candidate nuclei regions. Experimental results on 28 dierent histopathological images of skin tissue with dierent zooming factors show that the proposed technique provides a superior performance.

In this paper, we propose a robust technique for temporal alignment of video sequences with similar planar motions acquired using uncalibrated cameras. In this technique, we model the motion-based video temporal alignment problem as a spatio-temporal discrete trajectory point sets alignment problem. First, the trajectory of the object of interest is tracked throughout the videos. A probabilistic method is then developed to calculate the `soft' spatial correspondence between the trajectory point sets. Next, a dynamic time warping technique (DTW) is applied to the spatial correspondence information to compute the temporal alignment of the videos. The experimental results show that the proposed technique provides a superior performance over existing techniques for videos with similar trajectory patterns.

In this paper, a novel technique is proposed for temporal alignment of video sequences with similar motions acquired using uncalibrated cameras. In this technique, we model the motion-based video temporal alignment problem as a spatial-temporal discrete trajectory point sets alignment problem. The trajectory of the interested object is tracked though out the videos. A probabilistic method is then developed to calculate the spatial correspondence between trajectory point sets. Next, the dynamic time warping technique (DTW) is applied to the spatial correspondence information to compute the temporal alignment of the videos. The experimental results show that the proposed technique provides a superior performance over existing techniques for videos with planar motion.