Tissue segmentation in pathological images plays a crucial role for the diagnosis and prognosis of human cancers. However, due to the complexity of tumor micro-environment, it is difficult to annotate all tissue types especially for the categories with small tissue proportions, which limits the ability of the traditional tissue segmentation models to these tissue types with zero training samples. To address the above issues, we present a novel architecture, ZSPMLG, that relies on pathology vision-language foundation model (i.e., CONCH) to learn pixel-wise classifiers for both seen and unseen tissue types based on their text descriptions. Specifically, we firstly apply large language model (LLM) to generate the descriptions for both seen and unseen tissue categories, followed by feeding them to the CONCH text encoder to acquire their corresponding prototypes that are shared by both vision and semantic space. By considering that the textual descriptions of specific tissue categories can be observed from the pathological images at different scales of magnification, our ZSPMLG consists of Mixture of Local Experts (MoLE) and Mixture of Global Experts (MoGE) modules, where MoLE performs the specialized decoding that can map individual scale patch-level representation to dense pixel-level representation, while MoGE aims at fusing the multi-scale representations together. Finally, a convolutional layer is designed to map the pixel-level representation to the category prototype for tissue segmentation on both seen and unseen categories. We evaluate our method on three datasets and the experimental results demonstrate the superiority of our method on both seen and unseen tissue categories.

The recent advance of deep learning has shown great potential for nucleus detection which plays an important role in the histopathological examination. However, such accurate and reliable deep learning models usually need enough labeled data for training, which makes active learning an appealing learning paradigm to reduce the annotation efforts by experts. In open-set environments, active learning encounters the challenge that the unlabeled data usually contain non-target samples from the unknown classes, resulting in the failure of most active learning methods. Although active learning has been explored in many open-set classification tasks, research on active learning for nucleus detection in the open-set environment remains unexplored. To address the above issues, we propose a two-stage active learning framework designed for nucleus detection in the open-set environment (i.e., OpAL4ND). In the first stage, we propose a prototype-based query strategy based on the auxiliary detector to select a candidate set from known classes as pure as possible. In the second stage, we further query the most uncertain and representative samples from the candidate set for the nucleus detection task relying on the target detector. We evaluate the performance of our method on two nucleus detection datasets (i.e., the NuCLS and PanNuke datasets), and the experimental results indicate that our method can not only improve the selection quality on the known classes, but also achieve higher detection accuracy with lower annotation burden in comparison with the existing studies. Code is available at https://github.com/onbut/OpAL4ND.

Multi-modal ultrasound combines tissue information from multiple imaging perspectives, enabling more comprehensive lesion assessment. However, conventional multi-view learning methods typically assume uniform modality quality, ignoring variability caused by imaging noise and patient-specific factors. This oversight limits diagnostic reliability, especially when some modalities provide uncertain or conflicting information. To address this, we identify two key challenges in multi-modal ultrasound fusion: 1) how to quantify modality-wise uncertainty, and 2) how to resolve conflicts among predictions. We propose a novel method, termed TMUF (Trustworthy Multi-modal Ultrasound Fusion), which dynamically integrates information from different modalities through uncertainty calibration and conflict resolution. Specifically, we introduce a cross-modal uncertainty calibration regularizer to estimate evidence-based uncertainty across modalities, aligning uncertainty with prediction correctness. We further develop a credibility-aware fusion strategy that evaluates cross-modal consistency and uncertainty to distinguish credible from non-credible modalities, assigning fusion weights accordingly. We validate TMUF on public and private datasets for breast lesion and liver cancer diagnosis. The proposed method achieves diagnostic accuracies of 88.00% and 92.08%, respectively, outperforming state-of-the-art baselines. These results demonstrate the effectiveness of TMUF in enhancing diagnostic accuracy and robustness for multi-modal ultrasound.

Real-time tissue tracking is a fundamental task in liver ultrasound applications. Due to the periodic nature of liver motion, historical trajectories can offer valuable priors for target localization, particularly when foreground-background distinction is weak. However, existing trackers often exploit these trajectories as shortcuts, relying excessively on periodic respiratory patterns rather than true object appearance matching. In this work, we revisit liver tracking from a causal perspective and propose CUSTrack, a method that mitigates periodicity bias by decomposing and correcting the total causal effect of historical trajectories. We define periodicity bias as the direct causal effect of past states and eliminate it via counterfactual reasoning, preserving 'good' trajectory priors while suppressing 'bad' periodic bias. To ensure identifiability, we incorporate a deconfounding module that removes latent confounders from fused feature representations. Extensive experiments on liver ultrasound datasets demonstrate that CUSTrack achieves superior tracking accuracy and robustness under challenging conditions.

The rapid development of spatial transcriptomics (ST) allows researchers to measure the spatial-level gene expression in tissues. Although powerful, the cost for collecting the ST data is expensive, and thus several studies aim to predict gene expression in ST by utilizing their corresponding H/E stained pathology images. The existing ST based gene expression prediction models either adopt the pre-trained networks or rely on the handcrafted features to describe the pathology images, which still lack a systematic way to combine them together to define a spot-level representation that can reflect the topological profiles of different spots. On the other hand, all the ST based gene prediction models treat the prediction task for each gene independently, which overlook the fact that the exploration of potential interrelationships among them can help improve the prediction performance for individual genes. To address the above issues, we propose a multi-modal topology-embedded graph learning algorithm guided by prior Gene Ontology similarity information (i.e., M2TGLGO) to predict the spatial resolved genes from pathology images. Specifically, M2TGLGO co-learns the image representation of different spots from both deep and handcrafted features by considering the within-modal and inter-modal interactions. Next, to keep the topological structure among different spots, a spatial-oriented ranking module is also incorporated to preserve their neighborhood similarity information. Finally, we present a Gene Ontology knowledge guided graph neural network for simultaneously predicting multiple gene expressions by considering their functional associations. We evaluate our method on three public available ST datasets, the experimental results show the effectiveness of our M2TGLGO in comparison with the existing studies.

The integrative analysis of histopathological images and genomic data has received increasing attention for survival prediction of human cancers. However, the existing studies always hold the assumption that full modalities are available. As a matter of fact, the cost for collecting genomic data is high, which sometimes makes genomic data unavailable in testing samples. A common way of tackling such incompleteness is to generate the genomic representations from the pathology images. Nevertheless, such strategy still faces the following two challenges: (1) The gigapixel whole slide images (WSIs) are huge and thus hard for representation. (2) It is difficult to generate the genomic embeddings with diverse function categories in a unified generative framework. To address the above challenges, we propose a Conditional Latent Differentiation Variational AutoEncoder (LD-CVAE) for robust multimodal survival prediction, even with missing genomic data. Specifically, a Variational Information Bottleneck Transformer (VIBTrans) module is proposed to learn compressed pathological representations from the gigapixel WSIs. To generate different functional genomic features, we develop a novel Latent Differentiation Variational AutoEncoder (LD-VAE) to learn the genomic and function-specific posteriors for the genomic embeddings with diverse functions. Finally, we use the product-of-experts technique to integrate the genomic posterior and image posterior for the joint latent distribution estimation in LD-CVAE. We test the effectiveness of our method on five different cancer datasets, and the experimental results demonstrate its superiority in both complete and missing modality scenarios. The code is released†.

Prompt learning has emerged as a promising paradigm for adapting pre-trained vision-language models (VLMs) to few-shot whole slide image (WSI) classification by aligning visual features with textual representations, thereby reducing annotation cost and enhancing model generalization. Nevertheless, existing methods typically rely on slide-level prompts and fail to capture the subtype-specific phenotypic variations of histological entities (e.g., nuclei, glands) that are critical for cancer diagnosis. To address this gap, we propose Multi-scale Attribute-enhanced Prompt Learning (MAPLE), a hierarchical framework for few-shot WSI classification that jointly integrates multi-scale visual semantics and performs prediction at both the entity and slide levels. Specifically, we first leverage large language models (LLMs) to generate entity-level prompts that can help identify multi-scale histological entities and their phenotypic attributes, as well as slide-level prompts to capture global visual descriptions. Then, an entity-guided cross-attention module is proposed to generate entity-level features, followed by aligning with their corresponding subtype-specific attributes for fine-grained entity-level prediction. To enrich entity representations, we further develop a cross-scale entity graph learning module that can update these representations by capturing their semantic correlations within and across scales. The refined representations are then aggregated into a slide-level representation and aligned with the corresponding prompts for slide-level prediction. Finally, we combine both entity-level and slide-level outputs to produce the final prediction results. Results on three cancer cohorts confirm the effectiveness of our approach in addressing few-shot pathology diagnosis tasks.

The recent advance of deep learning technology brings the possibility of assisting the pathologist to predict the patients' survival from whole-slide pathological images (WSIs). However, most of the prevalent methods only worked on the sampled patches in specifically or randomly selected tumor areas of WSIs, which has very limited capability to capture the complex interactions between tumor and its surrounding micro-environment components. As a matter of fact, tumor is supported and nurtured in the heterogeneous tumor micro-environment(TME), and the detailed analysis of TME and their correlation with tumors are important to in-depth analyze the mechanism of cancer development. In this paper, we considered the spatial interactions among tumor and its two major TME components (i.e., lymphocytes and stromal fibrosis) and presented a Tumor Micro-environment Interactions Guided Graph Learning (TMEGL) algorithm for the prognosis prediction of human cancers. Specifically, we firstly selected different types of patches as nodes to build graph for each WSI. Then, a novel TME neighborhood organization guided graph embedding algorithm was proposed to learn node representations that can preserve their topological structure information. Finally, a Gated Graph Attention Network is applied to capture the survival-associated intersections among tumor and different TME components for clinical outcome prediction. We tested TMEGL on three cancer cohorts derived from The Cancer Genome Atlas (TCGA), and the experimental results indicated that TMEGL not only outperforms the existing WSI-based survival analysis models, but also has good explainable ability for survival prediction.

Dynamic contrast-enhanced ultrasound (CEUS) imaging can reflect the microvascular distribution and blood flow perfusion, thereby holding clinical significance in distinguishing between malignant and benign thyroid nodules. Notably, CEUS offers a meticulous visualization of the microvascular distribution surrounding the nodule, leading to an apparent increase in tumor size compared to gray-scale ultrasound (US). In the dual-image obtained, the lesion size enlarged from gray-scale US to CEUS, as the microvascular appeared to be continuously infiltrating the surrounding tissue. Although the infiltrative dilatation of microvasculature remains ambiguous, sonographers believe it may promote the diagnosis of thyroid nodules. We propose a deep learning model designed to emulate the diagnostic reasoning process employed by sonographers. This model integrates the observation of microvascular infiltration on dynamic CEUS, leveraging the additional insights provided by gray-scale US for enhanced diagnostic support. Specifically, temporal projection attention is implemented on time dimension of dynamic CEUS to represent the microvascular perfusion. Additionally, we employ a group of confidence maps with flexible Sigmoid Alpha Functions to aware and describe the infiltrative dilatation process. Moreover, a self-adaptive integration mechanism is introduced to dynamically integrate the assisted gray-scale US and the confidence maps of CEUS for individual patients, ensuring a trustworthy diagnosis of thyroid nodules. In this retrospective study, we collected a thyroid nodule dataset of 282 CEUS videos. The method achieves a superior diagnostic accuracy and sensitivity of 89.52% and 94.75%, respectively. These results suggest that imitating the diagnostic thinking of sonographers, encompassing dynamic microvascular perfusion and infiltrative expansion, proves beneficial for CEUS-based thyroid nodule diagnosis.

With the remarkable success of digital histopathology and the deep learning technology, many whole-slide pathological images (WSIs) based deep learning models are designed to help pathologists diagnose human cancers. Recently, rather than predicting categorical variables as in cancer diagnosis, several deep learning studies are also proposed to estimate the continuous variables such as the patients' survival or their transcriptional profile. However, most of the existing studies focus on conducting these predicting tasks separately, which overlooks the useful intrinsic correlation among them that can boost the prediction performance of each individual task. In addition, it is sill challenge to design the WSI-based deep learning models, since a WSI is with huge size but annotated with coarse label. In this study, we propose a general multi-instance multi-task learning framework (HistMIMT) for multi-purpose prediction from WSIs. Specifically, we firstly propose a novel multi-instance learning module (TMICS) considering both common and specific task information across different tasks to generate bag representation for each individual task. Then, a soft-mask based fusion module with channel attention (SFCA) is developed to leverage useful information from the related tasks to help improve the prediction performance on target task. We evaluate our method on three cancer cohorts derived from the Cancer Genome Atlas (TCGA). For each cohort, our multi-purpose prediction tasks range from cancer diagnosis, survival prediction and estimating the transcriptional profile of gene TP53. The experimental results demonstrated that HistMIMT can yield better outcome on all clinical prediction tasks than its competitors.

Medical image registration is a crucial preprocessing step in medical image processing. Due to the potential impact of treatments and disease progression on patients' organ geometry, such as in magnetic resonance imaging (MRI) and computed tomography (CT) images, medical image registration is of significant importance for cancer diagnosis, treatment planning, and therapy. Existing methods usually adopt convolutional neural networks and Transformer frameworks, but still face challenges to effectively solve both tissue robustness and registration accuracy. Therefore, we propose TSM, a hybrid Transformer-Convolution model, for non-rigid registration of volumetric medical multi-tissue images. We perform dynamic global filtering convolution in the frequency domain and multi-scale parallel convolution, capturing local tissue structure information; meanwhile, we use a special attention combination mechanism to obtain semantic associations among tissue structures. We evaluate our method on the publicly available LPBA40 and EMPIRE10 challenge datasets. With comparison of the state-of-the-arts, we improve the dice score by 1.2% on the LPBA40 dataset and by 3% on the EMPIRE10 dataset, achieving the best registration results.

Brain network , which characterizes the functional and structural interactions of brain regions with graph theory, has been widely utilized to diagnose brain diseases, such as autism spectrum disorder (ASD). It is a challenge to measure the network (or graph) similarity in brain network analysis . Graph kernel (i.e., kernel defined on graphs) offers an efficient tool for measuring the similarity of paired brain networks and yields the excellent classification performance in brain disease diagnosis. However, most of the existing graph kernels neglected the hierarchical architecture information of brain networks. To address this problem, in this paper, we propose an optimal transport based pyramid graph kernel for measuring brain network similarity and then apply it to brain disease classification. The main idea is to transform brain networks into pyramid structures, which reflect the hierarchical architecture information of the brain network with multi-resolution histograms. The optimal transport distance in pyramid structures is calculated for measuring transport costs between paired brain networks. Finally, the optimal transport based pyramid graph kernel is computed based on this optimal transport distance. To evaluate the effectiveness of the proposed optimal transport based pyramid graph kernel, the extensive experiments are performed in functional magnetic resonance imaging data of brain disease from the Autism Brain Imaging Data Exchange database. The experimental results show that our proposed optimal transport based pyramid graph kernel outperforms the state-of-the-art methods in ASD classification tasks .

Dynamic contrast-enhanced ultrasound (CEUS) imaging has been widely applied in lesion detection and characterization, due to its offered real-time observation of microvascular perfusion. Accurate lesion segmentation is of great importance to the quantitative and qualitative perfusion analysis. In this paper, we propose a novel dynamic perfusion representation and aggregation network (DpRAN) for the automatic segmentation of lesions using dynamic CEUS imaging. The core challenge of this work lies in enhancement dynamics modeling of various perfusion areas. Specifically, we divide enhancement features into the two scales: short-range enhancement patterns and long-range evolution tendency. To effectively represent real-time enhancement characteristics and aggregate them in a global view, we introduce the perfusion excitation (PE) gate and cross-attention temporal aggregation (CTA) module, respectively. Different from the common temporal fusion methods, we also introduce an uncertainty estimation strategy to assist the model to locate the critical enhancement point first, in which a relatively distinguished enhancement pattern is displayed. The segmentation performance of our DpRAN method is validated on our collected CEUS datasets of thyroid nodules. We obtain the mean dice coefficient (DSC) and intersection of union (IoU) of 0.794 and 0.676, respectively. Superior performance demonstrates its efficacy to capture distinguished enhancement characteristics for lesion recognition.

Contrast-enhanced ultrasound (CEUS) has emerged as a popular imaging modality in thyroid nodule diagnosis due to its ability to visualize vascular distribution in real time. Recently, a number of learning-based methods are dedicated to mine pathological-related enhancement dynamics and make prediction at one step, ignoring a native diagnostic dependency. In clinics, the differentiation of benign or malignant nodules always precedes the recognition of pathological types. In this paper, we propose a novel hierarchical temporal attention network (HiTAN) for thyroid nodule diagnosis using dynamic CEUS imaging, which unifies dynamic enhancement feature learning and hierarchical nodules classification into a deep framework. Specifically, this method decomposes the diagnosis of nodules into an ordered two-stage classification task, where diagnostic dependency is modeled by Gated Recurrent Units (GRUs). Besides, we design a local-to-global temporal aggregation (LGTA) operator to perform a comprehensive temporal fusion along the hierarchical prediction path. Particularly, local temporal information is defined as typical enhancement patterns identified with the guidance of perfusion representation learned from the differentiation level. Then, we leverage an attention mechanism to embed global enhancement dynamics into each identified salient pattern. In this study, we evaluate the proposed HiTAN method on the collected CEUS dataset of thyroid nodules. Extensive experimental results validate the efficacy of dynamic patterns learning, fusion and hierarchical diagnosis mechanism.

Generative Adversarial Networks (GANs) are powerful tools for reconstructing Compressed Sensing Magnetic Resonance Imaging (CS-MRI). However most recent works lack exploration of structure information of MRI images that is crucial for clinical diagnosis. To tackle this problem, we propose the Structure-Enhanced GAN (SEGAN) that aims at restoring structure information at both local and global scale. SEGAN defines a new structure regularization called Patch Correlation Regularization (PCR) which allows for efficient extraction of structure information. In addition, to further enhance the ability to uncover structure information, we propose a novel generator SU-Net by incorporating multiple-scale convolution filters into each layer. Besides, we theoretically analyze the convergence of stochastic factors contained in training process. Experimental results show that SEGAN is able to learn target structure information and achieves state-of-theart performance for CS-MRI reconstruction.