Over the past five years, artificial intelligence (AI) has introduced new models and methods for addressing the challenges associated with the broader adoption of AI models and systems in medicine. This paper reviews recent advances in AI for medical image and video analysis, outlines emerging paradigms, highlights pathways for successful clinical translation, and provides recommendations for future work. Hybrid Convolutional Neural Network (CNN) Transformer architectures now deliver state-of-the-art results in segmentation, classification, reconstruction, synthesis, and registration. Foundation and generative AI models enable the use of transfer learning to smaller datasets with limited ground truth. Federated learning supports privacy-preserving collaboration across institutions. Explainable and trustworthy AI approaches have become essential to foster clinician trust, ensure regulatory compliance, and facilitate ethical deployment. Together, these developments pave the way for integrating AI into radiology, pathology, and wider healthcare workflows.

Recently, deep neural networks have greatly advanced undersampled Magnetic Resonance Image (MRI) reconstruction, wherein most studies follow the one-anatomy-one-network fashion, i.e., each expert network is trained and evaluated for a specific anatomy. Apart from inefficiency in training multiple independent models, such convention ignores the shared de-aliasing knowledge across various anatomies which can benefit each other. To explore the shared knowledge, one naive way is to combine all the data from various anatomies to train an all-round network. Unfortunately, despite the existence of the shared de-aliasing knowledge, we reveal that the exclusive knowledge across different anatomies can deteriorate specific reconstruction targets, yielding overall performance degradation. Observing this, in this study, we present a novel deep MRI reconstruction framework with both anatomy-shared and anatomy-specific parameterized learners, aiming to "seek common ground while reserving differences" across different anatomies. Particularly, the primary anatomy-shared learners are exposed to different anatomies to model rich shared de-aliasing knowledge, while the efficient anatomy-specific learners are trained with their target anatomy for exclusive knowledge. Four different implementations of anatomy-specific learners are presented and explored on the top of our framework in two MRI reconstruction networks. Comprehensive experiments on brain, knee and cardiac MRI datasets demonstrate that three of these learners are able to enhance reconstruction performance via multiple anatomy collaborative learning. Extensive studies show that our strategy can also benefit multiple pulse sequence MRI reconstruction by integrating sequence-specific learners.

Immunoglobulins are crucial proteins produced by the immune system to identify and bind to foreign substances, playing an essential role in shielding organisms from infections and diseases. Designing specific antibodies opens new pathways for disease treatment. With the rise of deep learning, AI-driven drug design has become possible, leading to several methods for antibody design. However, many of these approaches require additional conditions that differ from real-world scenarios, making it challenging to incorporate them into existing antibody design processes. Here, we introduce IgGM, a generative model for the de novo design of immunoglobulins with functional specificity. IgGM simultaneously generates antibody sequences and structures for a given antigen, consisting of three core components: a pre-trained language model for extracting sequence features, a feature learning module for identifying pertinent features, and a prediction module that outputs designed antibody sequences and the predicted complete antibody-antigen complex structure. IgGM effectively predicts structures and designs novel antibodies and nanobodies. This makes it highly applicable in a wide range of practical situations related to antibody and nanobody design. Code is available at: https://github.com/TencentAI4S/IgGM.

Molecule-and-text cross-modal representation learning has emerged as a promising direction for enhancing the quality of molecular representation, thereby improving performance in various scientific fields. However, most approaches employ a global alignment approach to learn the knowledge from different modalities that may fail to capture fine-grained information, such as molecule-and-text fragments and stereoisomeric nuances, which is crucial for downstream tasks. Furthermore, it is incapable of modeling such information using a similar global alignment strategy due to the lack of annotations about the fine-grained fragments in the existing dataset. In this paper, we propose Atomas, a hierarchical molecular representation learning framework that jointly learns representations from SMILES strings and text. We design a Hierarchical Adaptive Alignment model to automatically learn the fine-grained fragment correspondence between two modalities and align these representations at three semantic levels. Atomas's end-to-end training framework supports understanding and generating molecules, enabling a wider range of downstream tasks. Atomas achieves superior performance across 12 tasks on 11 datasets, outperforming 11 baseline models thus highlighting the effectiveness and versatility of our method. Scaling experiments further demonstrate Atomas’s robustness and scalability. Moreover, visualization and qualitative analysis, validated by human experts, confirm the chemical relevance of our approach. Codes are released on ~\url{https://github.com/yikunpku/Atomas}.

Protein Language Models (PLMs), pre-trained on extensive evolutionary data from natural proteins, have emerged as indispensable tools for protein design. While powerful, PLMs often struggle to produce proteins with precisely specified functionalities or properties due to inherent challenges in controlling their outputs. In this work, we investigate the potential of Activation Steering, a technique originally developed for controlling text generation in Large Language Models (LLMs), to direct PLMs toward generating protein sequences with targeted properties. We propose a simple yet effective method that employs activation editing to steer PLM outputs, and extend this approach to protein optimization through a novel editing site identification module. Through comprehensive experiments on lysozyme-like sequence generation and optimization, we demonstrate that our methods can be seamlessly integrated into both auto-encoding and autoregressive PLMs without requiring additional training. These results highlight a promising direction for precise protein engineering using foundation models. Code is available at <https://github.com/Long-Kai/Steering-PLMs>.

Abstract Mutually exclusive gene expression, where gene pairs are expressed in strict alternation within individual cells, reflects fundamental inter-gene regulatory mechanisms and can reveal shifts in transcriptional programs during development or disease. Detecting such patterns is critical for resolving rare cellular subpopulations, temporally discrete states along pseudotime, and spatially segregated neighborhoods in single-cell and spatial multi-omics data. However, the sparsity and dropout inherent to single-cell data make mutually exclusive expression difficult to detect, leading conventional feature selection methods to overlook subtle yet functionally important genes. We present MULE, an unbiased framework that systematically organizes collective mutual exclusivity into a hierarchical taxonomy. Applying MULE to cardiac datasets, we uncovered robust upregulation of SPOCK1 and SLC6A6 in dilated cardiomyopathy, previously obscured by the inability to resolve pathological cardiomyocytes. In vivo and in vitro experiments demonstrated that stress-induced SLC6A6 upregulation serves a cardiomyocyte self-protective mechanism. Taurine supplementation reduced oxidative stress, restored calcium homeostasis, prevented cell death, and improved cardiac function post-injury. These findings elucidate a novel cardiomyocyte stress response and highlight the therapeutic promise of taurine supplementation for the treatment of dilated cardiomyopathy.

Learning with noisy labels (LNL) has attracted significant attention from the research community. Many recent LNL methods rely on the assumption that clean samples tend to have a "small loss." However, this assumption often fails to generalize to some real-world cases with imbalanced subpopulations, that is, training subpopulations that vary in sample size or recognition difficulty. Therefore, recent LNL methods face the risk of misclassifying those "informative" samples (e.g., hard samples or samples in the tail subpopulations) into noisy samples, leading to poor generalization performance. To address this issue, we propose a novel LNL method to deal with noisy labels and imbalanced subpopulations simultaneously. It first leverages sample correlation to estimate samples' clean probabilities for label correction and then utilizes corrected labels for distributionally robust optimization (DRO) to further improve the robustness. Specifically, in contrast to previous works using classification loss as the selection criterion, we introduce a feature-based metric that takes the sample correlation into account for estimating samples' clean probabilities. Then, we refurbish the noisy labels using the estimated clean probabilities and the pseudo-labels from the model's predictions. With refurbished labels, we use DRO to train the model to be robust to subpopulation imbalance. Extensive experiments on a wide range of benchmarks demonstrate that our technique can consistently improve state-of-the-art (SOTA) robust learning paradigms against noisy labels, especially when encountering imbalanced subpopulations. We provide our code in https://github.com/chenmc1996/LNL-IS.

One individual human’s immune repertoire consists of a huge set of adaptive immune receptors at a certain time point, representing the individual's adaptive immune state. Immune repertoire classification and associated receptor identification have the potential to make a transformative contribution to the development of novel vaccines and therapies. The vast number of instances and exceedingly low witness rate pose a great challenge to the immune repertoire classification, which can be formulated as a Massive Multiple Instance Learning (MMIL) problem. Traditional MIL methods, at both bag-level and instance-level, confront the issues of substantial computational burden or supervision ambiguity when handling massive instances. To address these issues, we propose a novel label disambiguation-based multimodal massive multiple instance learning approach (LaDM³IL) for immune repertoire classification. LaDM³IL adapts the instance-level MIL paradigm to deal with the issue of high computational cost and employs a specially-designed label disambiguation module for label correction, mitigating the impact of misleading supervision. To achieve a more comprehensive representation of each receptor, LaDM³IL leverages a multimodal fusion module with gating-based attention and tensor-fusion to integrate the information from gene segments and amino acid (AA) sequences of each immune receptor. Extensive experiments on the Cytomegalovirus (CMV) and Cancer datasets demonstrate the superior performance of the proposed LaDM³IL for both immune repertoire classification and associated receptor identification tasks. The code is publicly available at https://github.com/Josie-xufan/LaDM3IL.

Directed evolution, a cornerstone of protein optimization, is to harness natural mutational processes to enhance protein functionality. Existing Machine Learning-assisted Directed Evolution (MLDE) methodologies typically rely on data-driven strategies and often overlook the profound domain knowledge in biochemical fields. In this paper, we introduce a novel Knowledge-aware Reinforced Language Model (KnowRLM) for MLDE. An Amino Acid Knowledge Graph (AAKG) is constructed to represent the intricate biochemical relationships among amino acids. We further propose a Protein Language Model (PLM)-based policy network that iteratively samples mutants through preferential random walks on the AAKG using a dynamic sliding window mechanism. The novel mutants are actively sampled to fine-tune a fitness predictor as the reward model, providing feedback to the knowledge-aware policy. Finally, we optimize the whole system in an active learning approach that mimics biological settings in practice.KnowRLM stands out for its ability to utilize contextual amino acid information from knowledge graphs, thus attaining advantages from both statistical patterns of protein sequences and biochemical properties of amino acids.Extensive experiments demonstrate the superior performance of KnowRLM in more efficiently identifying high-fitness mutants compared to existing methods.

The decoder-only Transformer architecture with causal masking and relative position encoding (RPE) has become the de facto choice in language modeling. Despite its exceptional performance across various tasks, we have identified two limitations: First, it prevents all attended tokens from having zero weights during the softmax stage, even if the current embedding has sufficient self-contained information. This compels the model to assign disproportional excessive attention to specific tokens. Second, RPE-based Transformers are not universal approximators due to their limited capacity at encoding absolute positional information, which limits their application in position-critical tasks. In this work, we propose StableMask: a parameter-free method to address both limitations by refining the causal mask. It introduces pseudo-attention values to balance attention distributions and encodes absolute positional information via a progressively decreasing mask ratio. StableMask's effectiveness is validated both theoretically and empirically, showing significant enhancements in language models with parameter sizes ranging from 71M to 1.4B across diverse datasets and encoding methods. We further show that it supports integration with existing optimization techniques, making it easily usable in practical applications.

Incorporating Euclidean symmetries (e.g. rotation equivariance) as inductive biases into graph neural networks has improved their generalization ability and data efficiency in unbounded physical dynamics modeling. However, in various scientific and engineering applications, the symmetries of dynamics are frequently discrete due to the boundary conditions. Thus, existing GNNs either over-look necessary symmetry, resulting in suboptimal representation ability, or impose excessive equivariance, which fails to generalize to unobserved symmetric dynamics. In this work, we propose a general Discrete Equivariant Graph Neural Network (DEGNN) that guarantees equivariance to a given discrete point group. Specifically, we show that such discrete equivariant message passing could be constructed by transforming geometric features into permutation-invariant embeddings. Through relaxing continuous equivariant constraints, DEGNN can employ more geometric feature combinations to approximate unobserved physical object interaction functions. Two implementation approaches of DEGNN are proposed based on ranking or pooling permutation-invariant functions. We apply DEGNN to various physical dynamics, ranging from particle, molecular, crowd to vehicle dynamics. In twenty scenarios, DEGNN significantly outperforms existing state-of-the-art approaches. Moreover, we show that DEGNN is data efficient, learning with less data, and can generalize across scenarios such as unobserved orientation.

Protein optimization is a fundamental biological task aimed at enhancing theperformance of proteins by modifying their sequences. Computational methodsprimarily rely on evolutionary information (EI) encoded by protein languagemodels (PLMs) to predict fitness landscape for optimization. However, thesemethods suffer from a few limitations. (1) Evolutionary processes involve thesimultaneous consideration of multiple functional properties, often overshadowingthe specific property of interest. (2) Measurements of these properties tend to betailored to experimental conditions, leading to reduced generalizability of trainedmodels to novel proteins. To address these limitations, we introduce DenoisingProtein Language Models (DePLM), a novel approach that refines the evolutionaryinformation embodied in PLMs for improved protein optimization. Specifically, weconceptualize EI as comprising both property-relevant and irrelevant information,with the latter acting as “noise” for the optimization task at hand. Our approachinvolves denoising this EI in PLMs through a diffusion process conducted in therank space of property values, thereby enhancing model generalization and ensuringdataset-agnostic learning. Extensive experimental results have demonstrated thatDePLM not only surpasses the state-of-the-art in mutation effect prediction butalso exhibits strong generalization capabilities for novel proteins.

Spatially resolved transcriptomics (SRT) technologies facilitate gene expression profiling with spatial resolution in a naïve state. Nevertheless, current SRT technologies exhibit limitations, manifesting as either low transcript detection sensitivity or restricted gene throughput. These constraints result in diminished precision and coverage in gene measurement. In response, we introduce SpaGDA, a sophisticated deep learning-based graph domain adaptation framework for both scenarios of gene expression imputation and cell type identification in spatially resolved transcriptomics data by impartially transferring knowledge from reference scRNA-seq data. Systematic benchmarking analyses across several SRT datasets generated from different technologies have demonstrated SpaGDA's superior effectiveness compared to state-of-the-art methods in both scenarios. Further applied to three SRT datasets of different biological contexts, SpaGDA not only better recovers the well-established knowledge sourced from public atlases and existing scientific literature but also yields a more informative spatial expression pattern of genes. Together, these results demonstrate that SpaGDA can be used to overcome the challenges of current SRT data and provide more accurate insights into biological processes or disease development. The SpaGDA is available in https://github.com/shenrb/SpaGDA.

MOTIVATION: Cell-type clustering is a crucial first step for single-cell RNA-seq data analysis. However, existing clustering methods often provide different results on cluster assignments with respect to their own data pre-processing, choice of distance metrics, and strategies of feature extraction, thereby limiting their practical applications. RESULTS: We propose Cross-Tabulation Ensemble Clustering (CTEC) method that formulates two re-clustering strategies (distribution- and outlier-based) via cross-tabulation. Benchmarking experiments on five scRNA-Seq datasets illustrate that the proposed CTEC method offers significant improvements over the individual clustering methods. Moreover, CTEC-DB outperforms the state-of-the-art ensemble methods for single-cell data clustering, with 45.4% and 17.1% improvement over the single-cell aggregated from ensemble clustering method (SAFE) and the single-cell aggregated clustering via Mixture model ensemble method (SAME), respectively, on the two-method ensemble test. AVAILABILITY AND IMPLEMENTATION: The source code of the benchmark in this work is available at the GitHub repository https://github.com/LWCHN/CTEC.git.

Cryo-EM in single particle analysis is known to have low SNR and requires to utilize several frames of the same particle sample to restore one high-quality image for visualizing that particle. However, the low SNR of cryo-EM movie and motion caused by beam striking make the task very challenging. Video enhancement algorithms in computer vision shed new light on tackling such tasks by utilizing deep neural networks. However, they are designed for natural images with high SNR. Meanwhile, the lack of ground truth in cryo-EM movie seems to be one major limiting factor of the progress. Hence, we present a synthetic cryo-EM movie generation pipeline, which can produce realistic diverse cryo-EM movie datasets with low-SNR movie frames and multiple ground truth values. Then we propose a deep spatio-temporal network (DST-Net) for cryo-EM movie frame enhancement trained on our synthetic data. Spatial and temporal features are first extracted from each frame. Spatio-temporal fusion and high-resolution re-constructor are designed to obtain the enhanced output. For evaluation, we train our model on seven synthetic cryo-EM movie datasets and infer on real cryo-EM data. The experimental results show that DST-Net can achieve better enhancement performance both quantitatively and qualitatively compared with others.

The in silico prediction of antigen specificity in adaptive immune receptors (AIRs), such as T-cell receptors (TCRs), is essential for understanding immunological processes and developing targeted therapies. The V(D)J gene rearrangement is a critical biological process that generates diversity in amino acid (AA) sequences in antigen-binding regions, enabling AIRs to recognize a wide range of antigens from various pathogens and "altered self cells" observed in cancers. The huge diversity of AIRs presents a significant challenge to existing computational methods for antigen specificity prediction. To address these complexities, we introduce Multimodal-AIR-BERT, a novel multimodal pre-trained model aimed at enhancing the prediction of antigen-binding specificity in TCRs. It comprises a pre-trained sequence encoder, a gene encoder, and a multimodal fusion module with gating-based attention and tensor fusion to calibrate and integrate the V(D)J gene and AA sequence features of TCRs, thereby generating more informative representations. The integration of V(D)J gene information, which provides insights often unobtainable from sequences alone, benefits Multimodal-AIR-BERT in performance enhancement compared to its sequence-modality-only counterpart. Collectively, our work provides an advancement in the accurate prediction of antigen-binding specificity. As the precision of this specificity prediction improves, it can potentially pave the way for targeted immune therapies and deeper insights into the interactions within the immune system.

Immune repertoire classification, a typical multiple instance learning (MIL) problem, is a frontier research topic in computational biology that makes transformative contributions to new vaccines and immune therapies. However, the traditional instance-space MIL, directly assigning bag-level labels to instances, suffers from the massive amount of noisy labels and extremely low witness rate. In this work, we propose a noisy-label-learning formulation to solve the immune repertoire classification task. To remedy the inaccurate supervision of repertoire-level labels for a sequence-level classifier, we design a robust training strategy: The initial labels are smoothed to be asymmetric and are progressively corrected using the model's predictions throughout the training process. Furthermore, two models with the same architecture but different parameter initialization are co-trained simultaneously to remedy the known ``confirmation bias'' problem in the self-training-like schema. As a result, we obtain accurate sequence-level classification and, subsequently, repertoire-level classification. Experiments on the Cytomegalovirus (CMV) and Cancer datasets demonstrate our method's effectiveness and superior performance on sequence-level and repertoire-level tasks. Code available at https://github.com/TencentAILabHealthcare/NLL-IRC.

Technological advances have now made it possible to simultaneously profile the changes of epigenomic, transcriptomic and proteomic at the single cell level, allowing a more unified view of cellular phenotypes and heterogeneities. However, current computational tools for single-cell multi-omics data integration are mainly tailored for bi-modality data, so new tools are urgently needed to integrate tri-modality data with complex associations. To this end, we develop scMHNN to integrate single-cell multi-omics data based on hypergraph neural network. After modeling the complex data associations among various modalities, scMHNN performs message passing process on the multi-omics hypergraph, which can capture the high-order data relationships and integrate the multiple heterogeneous features. Followingly, scMHNN learns discriminative cell representation via a dual-contrastive loss in self-supervised manner. Based on the pretrained hypergraph encoder, we further introduce the pre-training and fine-tuning paradigm, which allows more accurate cell-type annotation with only a small number of labeled cells as reference. Benchmarking results on real and simulated single-cell tri-modality datasets indicate that scMHNN outperforms other competing methods on both cell clustering and cell-type annotation tasks. In addition, we also demonstrate scMHNN facilitates various downstream tasks, such as cell marker detection and enrichment analysis.

Antimicrobial peptides (AMPs) are short peptides that play crucial roles in diverse biological processes and have various functional activities against target organisms. Due to the abuse of chemical antibiotics and microbial pathogens' increasing resistance to antibiotics, AMPs have the potential to be alternatives to antibiotics. As such, the identification of AMPs has become a widely discussed topic. A variety of computational approaches have been developed to identify AMPs based on machine learning algorithms. However, most of them are not capable of predicting the functional activities of AMPs, and those predictors that can specify activities only focus on a few of them. In this study, we first surveyed 10 predictors that can identify AMPs and their functional activities in terms of the features they employed and the algorithms they utilized. Then, we constructed comprehensive AMP datasets and proposed a new deep learning-based framework, iAMPCN (identification of AMPs based on CNNs), to identify AMPs and their related 22 functional activities. Our experiments demonstrate that iAMPCN significantly improved the prediction performance of AMPs and their corresponding functional activities based on four types of sequence features. Benchmarking experiments on the independent test datasets showed that iAMPCN outperformed a number of state-of-the-art approaches for predicting AMPs and their functional activities. Furthermore, we analyzed the amino acid preferences of different AMP activities and evaluated the model on datasets of varying sequence redundancy thresholds. To facilitate the community-wide identification of AMPs and their corresponding functional types, we have made the source codes of iAMPCN publicly available at https://github.com/joy50706/iAMPCN/tree/master. We anticipate that iAMPCN can be explored as a valuable tool for identifying potential AMPs with specific functional activities for further experimental validation.

Underlying medical conditions, such as cancer, kidney disease and heart failure, are associated with a higher risk for severe COVID-19. Accurate classification of COVID-19 patients with underlying medical conditions is critical for personalized treatment decision and prognosis estimation. In this study, we propose an interpretable artificial intelligence model termed VDJMiner to mine the underlying medical conditions and predict the prognosis of COVID-19 patients according to their immune repertoires. In a cohort of more than 1400 COVID-19 patients, VDJMiner accurately identifies multiple underlying medical conditions, including cancers, chronic kidney disease, autoimmune disease, diabetes, congestive heart failure, coronary artery disease, asthma and chronic obstructive pulmonary disease, with an average area under the receiver operating characteristic curve (AUC) of 0.961. Meanwhile, in this same cohort, VDJMiner achieves an AUC of 0.922 in predicting severe COVID-19. Moreover, VDJMiner achieves an accuracy of 0.857 in predicting the response of COVID-19 patients to tocilizumab treatment on the leave-one-out test. Additionally, VDJMiner interpretively mines and scores V(D)J gene segments of the T-cell receptors that are associated with the disease. The identified associations between single-cell V(D)J gene segments and COVID-19 are highly consistent with previous studies. The source code of VDJMiner is publicly accessible at https://github.com/TencentAILabHealthcare/VDJMiner. The web server of VDJMiner is available at https://gene.ai.tencent.com/VDJMiner/.

Accurately predicting the antigen-binding specificity of adaptive immune receptors (AIRs), such as T-cell receptors (TCRs) and B-cell receptors (BCRs), is essential for discovering new immune therapies. However, the diversity of AIR chain sequences limits the accuracy of current prediction methods. This study introduces SC-AIR-BERT, a pre-trained model that learns comprehensive sequence representations of paired AIR chains to improve binding specificity prediction. SC-AIR-BERT first learns the 'language' of AIR sequences through self-supervised pre-training on a large cohort of paired AIR chains from multiple single-cell resources. The model is then fine-tuned with a multilayer perceptron head for binding specificity prediction, employing the K-mer strategy to enhance sequence representation learning. Extensive experiments demonstrate the superior AUC performance of SC-AIR-BERT compared with current methods for TCR- and BCR-binding specificity prediction.

MOTIVATION: Multiple instance learning (MIL) is a powerful technique to classify whole slide images (WSIs) for diagnostic pathology. The key challenge of MIL on WSI classification is to discover the critical instances that trigger the bag label. However, tumor heterogeneity significantly hinders the algorithm's performance. RESULTS: Here, we propose a novel multiplex-detection-based multiple instance learning (MDMIL) which targets tumor heterogeneity by multiplex detection strategy and feature constraints among samples. Specifically, the internal query generated after the probability distribution analysis and the variational query optimized throughout the training process are utilized to detect potential instances in the form of internal and external assistance, respectively. The multiplex detection strategy significantly improves the instance-mining capacity of the deep neural network. Meanwhile, a memory-based contrastive loss is proposed to reach consistency on various phenotypes in the feature space. The novel network and loss function jointly achieve high robustness towards tumor heterogeneity. We conduct experiments on three computational pathology datasets, e.g. CAMELYON16, TCGA-NSCLC, and TCGA-RCC. Benchmarking experiments on the three datasets illustrate that our proposed MDMIL approach achieves superior performance over several existing state-of-the-art methods. AVAILABILITY AND IMPLEMENTATION: MDMIL is available for academic purposes at https://github.com/ZacharyWang-007/MDMIL.

Dense prediction in medical volume provides enriched guidance for clinical analysis. CNN backbones have met bottleneck due to lack of long-range dependencies and global context modeling power. Recent works proposed to combine vision transformer with CNN, due to its strong global capture ability and learning capability. However, most works are limited to simply applying pure transformer with several fatal flaws (i.e., lack of inductive bias, heavy computation and little consideration for 3D data). Therefore, designing an elegant and efficient vision transformer learner for dense prediction in medical volume is promising and challenging. In this paper, we propose a novel 3D Shuffle-Mixer network of a new Local Vision Transformer-MLP paradigm for medical dense prediction. In our network, a local vision transformer block is utilized to shuffle and learn spatial context from full-view slices of rearranged volume, a residual axial-MLP is designed to mix and capture remaining volume context in a slice-aware manner, and a MLP view aggregator is employed to project the learned full-view rich context to the volume feature in a view-aware manner. Moreover, an Adaptive Scaled Enhanced Shortcut is proposed for local vision transformer to enhance feature along spatial and channel dimensions adaptively, and a CrossMerge is proposed to skip-connect the multi-scale feature appropriately in the pyramid architecture. Extensive experiments demonstrate the proposed model outperforms other state-of-the-art medical dense prediction methods.

Inferring gene regulatory networks based on single-cell transcriptomes is critical for systematically understanding cell-specific regulatory networks and discovering drug targets in tumor cells. Here we show that existing methods mainly perform co-expression analysis and apply the image-based model to deal with the non-euclidean scRNA-seq data, which may not reasonably handle the dropout problem and not fully take advantage of the validated gene regulatory topology. We propose a graph-based end-to-end deep learning model for GRN inference (GRNInfer) with the help of known regulatory relations through transductive learning. The robustness and superiority of the model are demonstrated by comparative experiments.

Integration of heterogeneous and high-dimensional data (e.g., multiomics) is becoming increasingly important. Existing multimodal classification algorithms mainly focus on improving performance by exploiting the complementarity from different modalities. However, conventional approaches are basically weak in providing trustworthy multimodal fusion, especially for safety-critical applications (e.g., medical diagnosis). For this issue, we propose a novel trustworthy multimodal classification algorithm termed Multimodal Dynamics, which dynamically evaluates both the feature-level and modality-level informativeness for different samples and thus trustworthily integrates multiple modalities. Specifically, a sparse gating is introduced to capture the information variation of each within-modality feature and the true class probability is employed to assess the classification confidence of each modality. Then a transparent fusion algorithm based on the dynamical informativeness estimation strategy is induced. To the best of our knowledge, this is the first work to jointly model both feature and modality variation for different samples to provide trustworthy fusion in multi-modal classification. Extensive experiments are conducted on multimodal medical classification datasets. In these experiments, superior performance and trustworthiness of our algorithm are clearly validated compared to the state-of-the-art methods.

Morphology of neuron trees is a key indicator to delineate neuronal cell-types, analyze brain development process, and evaluate pathological changes in neurological diseases. Traditional analysis mostly relies on heuristic features and visual inspections. A quantitative, informative, and comprehensive representation of neuron morphology is largely absent but desired. To fill this gap, in this work, we adopt a Tree-LSTM network to encode neuron morphology and introduce a self-supervised learning framework named TreeMoCo to learn features without the need for labels. We test TreeMoCo on 2403 high-quality 3D neuron reconstructions of mouse brains from three different public resources. Our results show that TreeMoCo is effective in both classifying major brain cell-types and identifying sub-types. To our best knowledge, TreeMoCo is the very first to explore learning the representation of neuron tree morphology with contrastive learning. It has a great potential to shed new light on quantitative neuron morphology analysis. Code is available at https://github.com/TencentAILabHealthcare/NeuronRepresentation.

Subpopulation shift widely exists in many real-world machine learning applications, referring to the training and test distributions containing the same subpopulation groups but varying in subpopulation frequencies. Importance reweighting is a normal way to handle the subpopulation shift issue by imposing constant or adaptive sampling weights on each sample in the training dataset. However, some recent studies have recognized that most of these approaches fail to improve the performance over empirical risk minimization especially when applied to over-parameterized neural networks. In this work, we propose a simple yet practical framework, called uncertainty-aware mixup (UMIX), to mitigate the overfitting issue in over-parameterized models by reweighting the ''mixed'' samples according to the sample uncertainty. The training-trajectories-based uncertainty estimation is equipped in the proposed UMIX for each sample to flexibly characterize the subpopulation distribution. We also provide insightful theoretical analysis to verify that UMIX achieves better generalization bounds over prior works. Further, we conduct extensive empirical studies across a wide range of tasks to validate the effectiveness of our method both qualitatively and quantitatively. Code is available at https://github.com/TencentAILabHealthcare/UMIX.

Subcellular localization of messenger RNAs (mRNAs) plays a key role in the spatial regulation of gene activity. The functions of mRNAs have been shown to be closely linked with their localizations. As such, understanding of the subcellular localizations of mRNAs can help elucidate gene regulatory networks. Despite several computational methods that have been developed to predict mRNA localizations within cells, there is still much room for improvement in predictive performance, especially for the multiple-location prediction. In this study, we proposed a novel multi-label multi-class predictor, termed Clarion, for mRNA subcellular localization prediction. Clarion was developed based on a manually curated benchmark dataset and leveraged the weighted series method for multi-label transformation. Extensive benchmarking tests demonstrated Clarion achieved competitive predictive performance and the weighted series method plays a crucial role in securing superior performance of Clarion. In addition, the independent test results indicate that Clarion outperformed the state-of-the-art methods and can secure accuracy of 81.47, 91.29, 79.77, 92.10, 89.15, 83.74, 80.74, 79.23 and 84.74% for chromatin, cytoplasm, cytosol, exosome, membrane, nucleolus, nucleoplasm, nucleus and ribosome, respectively. The webserver and local stand-alone tool of Clarion is freely available at http://monash.bioweb.cloud.edu.au/Clarion/.

MOTIVATION: Advanced deep learning techniques have been widely applied in disease diagnosis and prognosis with clinical omics, especially gene expression data. In the regulation of biological processes and disease progression, genes often work interactively rather than individually. Therefore, investigating gene association information and co-functional gene modules can facilitate disease state prediction. RESULTS: To explore the gene modules and inter-gene relational information contained in the omics data, we propose a novel multi-level attention graph neural network (MLA-GNN) for disease diagnosis and prognosis. Specifically, we format omics data into co-expression graphs via weighted correlation network analysis, and then construct multi-level graph features, finally fuse them through a well-designed multi-level graph feature fully fusion module to conduct predictions. For model interpretation, a novel full-gradient graph saliency mechanism is developed to identify the disease-relevant genes. MLA-GNN achieves state-of-the-art performance on transcriptomic data from TCGA-LGG/TCGA-GBM and proteomic data from coronavirus disease 2019 (COVID-19)/non-COVID-19 patient sera. More importantly, the relevant genes selected by our model are interpretable and are consistent with the clinical understanding. AVAILABILITYAND IMPLEMENTATION: The codes are available at https://github.com/TencentAILabHealthcare/MLA-GNN.

Brain midline delineation plays an important role in guiding intracranial hemorrhage surgery, which still remains a challenging task since hemorrhage shifts the normal brain configuration. Most previous studies detected brain midline on 2D plane and did not handle hemorrhage cases well. We propose a novel and efficient hemisphere-segmentation framework (HSF) for 3D brain midline surface delineation. Specifically, we formulate the brain midline delineation as a 3D hemisphere segmentation task, and employ an edge detector and a smooth regularization loss to generate the midline surface. We also introduce a distance-weighted map to keep the attention on the midline. Furthermore, we adopt rectification learning to handle various head poses. Finally, considering the complex situation of ventricle break-in for hemorrhages in bilateral intraventricular (B-IVH) cases, we identify those cases via a classification model and design a midline correction strategy to locally adjust the midline. To our best knowledge, it is the first study focusing on delineating the brain midline surface on 3D CT images of hemorrhage patients and handling the situation of ventricle break-in. Extensive validation on our large in-house datasets (519 patients) and the public CQ500 dataset (491 patients), demonstrates that our method outperforms state-of-the-art methods on brain midline delineation.

Clinical omics, especially gene expression data, have been widely studied and successfully applied for disease diagnosis using machine learning techniques. As genes often work interactively rather than individually, investigating co-functional gene modules can improve our understanding of disease mechanisms and facilitate disease state prediction. To this end, we in this paper propose a novel Multi-Level Enhanced Graph ATtention (MLE-GAT) network to explore the gene modules and intergene relational information contained in the omics data. In specific, we first format the omics data of each patient into co-expression graphs using weighted correlation network analysis (WGCNA) and then feed them to a well-designed multi-level graph feature fully fusion (MGFFF) module for disease diagnosis. For model interpretation, we develop a novel full-gradient graph saliency (FGS) mechanism to identify the disease-relevant genes. Comprehensive experiments show that our proposed MLE-GAT achieves state-of-the-art performance on transcriptomics data from TCGA-LGG/TCGA-GBM and proteomics data from COVID-19/non-COVID-19 patient sera.

The emerging COVID-19 variants lead to a new wave of infections, spreading more rapidly with more severe illnesses. The adaptive immune system plays an essential role in the control and clearance of viral infection and influences clinical outcomes. However, the understanding of the adaptive immune responses to COVID-19 is not sufficient, which impedes the development progress of treatments and vaccines. To address this issue, we proposed a machine-learning-based method (termed as VDJ-Seg-Miner) to mine the underlying associations between the V(D)J gene segments of the T cell receptor in personalized immune repertoires and COVID-19 disease characteristics for immune system analysis. Our VDJ-Seg-Miner can interpretively reveal multiple associations between the V(D)J gene segments and COVID-19 disease characteristics and assign confidence scores to indicate its confidence in each revealed association. Furthermore, experimental results based on the real-world dataset suggested that the identified associations were highly consistent with those reported in previous work.

Metastatic involvement of lymph nodes is one of the most important prognostic variables for many cancers. Several deep learning based algorithms have been developed to segment metastatic regions in pathological images to help predict prognosis. However, the training of these methods requires a large amount of annotated data, and the labeling task is an extremely time-consuming process for human annotators. In order to reduce the annotation burden, we for the first time propose a semi-automatic annotation method (PiPo-Net) for the labeling of pathological images. The method is comprised of two subnetworks, a pixel-wise segmentation network (Pi-Net) and a polygon-based annotation network (Po-Net). The Pi-Net adopts an improved encoder-decoder architecture and can effectively aggregate multi-scale image features. The Po-Net is built on the Pi-Net and leverages a two-layer recurrent neural network to generate tight-bounded polygons for the metastatic regions. Corresponding to the proposed network architecture, a loss function called PiPo-loss is introduced to help optimize the whole network. The main advantage of our method is that it integrates human annotators into the prediction loop, allowing to iteratively refine the predictions according to the suggestions from human annotators. We evaluate our method on Camelyon16 database and achieve a Dice score of 91% in the initial annotation attempt. We also demonstrate the effectiveness of the human-network collaborative annotation, which achieves promising labeling results, verifying the advantages of our proposed method.

Multiple instance learning (MIL) is a typical weakly-supervised learning method where the label is associated with a bag of instances instead of a single instance. Despite extensive research over past years, effectively deploying MIL remains an open and challenging problem, especially when the commonly assumed standard multiple instance (SMI) assumption is not satisfied. In this paper, we propose a multiple instance learning method based on deep graph convolutional network and feature selection (FS-GCN-MIL) for histopathological image classification. The proposed method consists of three components, including instance-level feature extraction, instance-level feature selection, and bag-level classification. We develop a self-supervised learning mechanism to train the feature extractor based on a combination model of variational autoencoder and generative adversarial network (VAE-GAN). Additionally, we propose a novel instance-level feature selection method to select the discriminative instance features. Furthermore, we employ a graph convolutional network (GCN) for learning the bag-level representation and then performing the classification. We apply the proposed method in the prediction of lymph node metastasis using histopathological images of colorectal cancer. Experimental results demonstrate that the proposed method achieves superior performance compared to the state-of-the-art methods.

Prognostic tumor growth modeling via volumetric medical imaging observations can potentially lead to better outcomes of tumor treatment management and surgical planning. Recent advances of convolutional networks (ConvNets) have demonstrated higher accuracy than traditional mathematical models can be achieved in predicting future tumor volumes. This indicates that deep learning based data-driven techniques may have great potentials on addressing such problem. However, current 2D image patch based modeling approaches can not make full use of the spatio-temporal imaging context of the tumor's longitudinal 4D (3D + time) patient data. Moreover, they are incapable to predict clinically-relevant tumor properties, other than the tumor volumes. In this paper, we exploit to formulate the tumor growth process through convolutional Long Short-Term Memory (ConvLSTM) that extract tumor's static imaging appearances and simultaneously capture its temporal dynamic changes within a single network. We extend ConvLSTM into the spatio-temporal domain (ST-ConvLSTM) by jointly learning the inter-slice 3D contexts and the longitudinal or temporal dynamics from multiple patient studies. Our approach can incorporate other non-imaging patient information in an end-to-end trainable manner. Experiments are conducted on the largest 4D longitudinal tumor dataset of 33 patients to date. Results validate that the proposed ST-ConvLSTM model produces a Dice score of 83.2%±5.1% and a RVD of 11.2%±10.8%, both statistically significantly outperforming (p < 0.05) other compared methods of traditional linear model, ConvLSTM, and generative adversarial network (GAN) under the metric of predicting future tumor volumes. Additionally, our new method enables the prediction of both cell density and CT intensity numbers. Last, we demonstrate the generalizability of ST-ConvLSTM by employing it in 4D medical image segmentation task, which achieves an averaged Dice score of 86.3%±1.2% for left-ventricle segmentation in 4D ultrasound with 3 seconds per patient case.

Tumor growth is associated with cell invasion and mass-effect, which are traditionally formulated by mathematical models, namely reaction-diffusion equations and biomechanics. Such models can be personalized based on clinical measurements to build the predictive models for tumor growth. In this paper, we investigate the possibility of using deep convolutional neural networks to directly represent and learn the cell invasion and mass-effect, and to predict the subsequent involvement regions of a tumor. The invasion network learns the cell invasion from information related to metabolic rate, cell density, and tumor boundary derived from multimodal imaging data. The expansion network models the mass-effect from the growing motion of tumor mass. We also study different architectures that fuse the invasion and expansion networks, in order to exploit the inherent correlations among them. Our network can easily be trained on population data and personalized to a target patient, unlike most previous mathematical modeling methods that fail to incorporate population data. Quantitative experiments on a pancreatic tumor data set show that the proposed method substantially outperforms a state-of-the-art mathematical model-based approach in both accuracy and efficiency, and that the information captured by each of the two subnetworks is complementary.

The recent rapid and tremendous success of deep convolutional neural networks (CNN) on many challenging computer vision tasks largely derives from the accessibility of the well-annotated ImageNet and PASCAL VOC datasets. Nevertheless, unsupervised image categorization (i.e., without the ground-truth labeling) is much less investigated, yet critically important and difficult when annotations are extremely hard to obtain in the conventional way of "Google Search" and crowd sourcing. We address this problem by presenting a looped deep pseudo-task optimization (LDPO) framework for joint mining of deep CNN features and image labels. Our method is conceptually simple and rests upon the hypothesized "convergence" of better labels leading to better trained CNN models which in turn feed more discriminative image representations to facilitate more meaningful clusters/labels. Our proposed method is validated in tackling two important applications: 1) Large-scale medical image annotation has always been a prohibitively expensive and easily-biased task even for well-trained radiologists. Significantly better image categorization results are achieved via our proposed approach compared to the previous state-of-the-art method. 2) Unsupervised scene recognition on representative and publicly available datasets with our proposed technique is examined. The LDPO achieves excellent quantitative scene classification results. On the MIT indoor scene dataset, it attains a clustering accuracy of 75:3%, compared to the state-of-the-art supervised classification accuracy of 81:0% (when both are based on the VGG-VD model).

Automation-assisted cervical screening via Pap smear or liquid-based cytology (LBC) is a highly effective cell imaging based cancer detection tool, where cells are partitioned into "abnormal" and "normal" categories. However, the success of most traditional classification methods relies on the presence of accurate cell segmentations. Despite sixty years of research in this field, accurate segmentation remains a challenge in the presence of cell clusters and pathologies. Moreover, previous classification methods are only built upon the extraction of hand-crafted features, such as morphology and texture. This paper addresses these limitations by proposing a method to directly classify cervical cells-without prior segmentation-based on deep features, using convolutional neural networks (ConvNets). First, the ConvNet is pretrained on a natural image dataset. It is subsequently fine-tuned on a cervical cell dataset consisting of adaptively resampled image patches coarsely centered on the nuclei. In the testing phase, aggregation is used to average the prediction scores of a similar set of image patches. The proposed method is evaluated on both Pap smear and LBC datasets. Results show that our method outperforms previous algorithms in classification accuracy (98.3%), area under the curve (0.99) values, and especially specificity (98.3%), when applied to the Herlev benchmark Pap smear dataset and evaluated using five-fold cross validation. Similar superior performances are also achieved on the HEMLBC (H&E stained manual LBC) dataset. Our method is promising for the development of automation-assisted reading systems in primary cervical screening.

Pancreatic neuroendocrine tumors are abnormal growths of hormone-producing cells in the pancreas. Unlike the brain which is protected by the skull, the pancreas can be significantly deformed by its surrounding organs. Consequently, the tumor shape differences observable from images at different time points arise from both tumor growth and pancreatic motion, and tumor growth model personalization may be compromised if such motion is ignored. Therefore, we incorporate pancreatic motion information derived from deformable image registration in model personalization. For more accurate mechanical interactions between tumor growth and pancreatic motion, elastic-growth decomposition is used with a hyperelastic constitutive law to model the mass effect, which allows growth modeling while conserving the mechanical properties. Furthermore, a way of coupling the finite difference method and the finite element method is proposed to greatly reduce the computation time. With both 2-[18F]-fluoro-2-deoxy-D-glucose positron emission tomographic and contrast-enhanced computed tomographic images, functional, structural, and motion data are combined for a patient-specific model. Experiments on synthetic and clinical data show the importance of image-derived motion on estimating pathophysiologically plausible mechanical properties and the promising performance of our framework. From seven patient data sets, the recall, precision, Dice coefficient, relative volume difference, and average surface distance between the personalized tumor growth simulations and the measurements were 83.2 ±8.8%, 86.9 ±8.3%, 84.4 ±4.0%, 13.9 ±9.8%, and 0.6 ±0.1 mm, respectively.

Despite the recent advances in automatically describing image contents, their applications have been mostly limited to image caption datasets containing natural images (e.g., Flickr 30k, MSCOCO). In this paper, we present a deep learning model to efficiently detect a disease from an image and annotate its contexts (e.g., location, severity and the affected organs). We employ a publicly available radiology dataset of chest x-rays and their reports, and use its image annotations to mine disease names to train convolutional neural networks (CNNs). In doing so, we adopt various regularization techniques to circumvent the large normalvs-diseased cases bias. Recurrent neural networks (RNNs) are then trained to describe the contexts of a detected disease, based on the deep CNN features. Moreover, we introduce a novel approach to use the weights of the already trained pair of CNN/RNN on the domain-specific image/text dataset, to infer the joint image/text contexts for composite image labeling. Significantly improved image annotation results are demonstrated using the recurrent neural cascade model by taking the joint image/text contexts into account.

Despite tremendous progress in computer vision, there has not been an attempt to apply machine learning on very large-scale medical image databases. We present an interleaved text/image deep learning system to extract and mine the semantic interactions of radiology images and reports from a national research hospital's Picture Archiving and Communication System. With natural language processing, we mine a collection of $\sim$216K representative two-dimensional images selected by clinicians for diagnostic reference and match the images with their descriptions in an automated manner. We then employ a weakly supervised approach using all of our available data to build models for generating approximate interpretations of patient images. Finally, we demonstrate a more strictly supervised approach to detect the presence and absence of a number of frequent disease types, providing more specific interpretations of patient scans. A relatively small amount of data is used for this part, due to the challenge in gathering quality labels from large raw text data. Our work shows the feasibility of large-scale learning and prediction in electronic patient records available in most modern clinical institutions. It also demonstrates the trade-offs to consider in designing machine learning systems for analyzing large medical data.

Automated computer-aided detection (CADe) has been an important tool in clinical practice and research. State-of-the-art methods often show high sensitivities at the cost of high false-positives (FP) per patient rates. We design a two-tiered coarse-to-fine cascade framework that first operates a candidate generation system at sensitivities ∼ 100% of but at high FP levels. By leveraging existing CADe systems, coordinates of regions or volumes of interest (ROI or VOI) are generated and function as input for a second tier, which is our focus in this study. In this second stage, we generate 2D (two-dimensional) or 2.5D views via sampling through scale transformations, random translations and rotations. These random views are used to train deep convolutional neural network (ConvNet) classifiers. In testing, the ConvNets assign class (e.g., lesion, pathology) probabilities for a new set of random views that are then averaged to compute a final per-candidate classification probability. This second tier behaves as a highly selective process to reject difficult false positives while preserving high sensitivities. The methods are evaluated on three data sets: 59 patients for sclerotic metastasis detection, 176 patients for lymph node detection, and 1,186 patients for colonic polyp detection. Experimental results show the ability of ConvNets to generalize well to different medical imaging CADe applications and scale elegantly to various data sets. Our proposed methods improve performance markedly in all cases. Sensitivities improved from 57% to 70%, 43% to 77%, and 58% to 75% at 3 FPs per patient for sclerotic metastases, lymph nodes and colonic polyps, respectively.

Remarkable progress has been made in image recognition, primarily due to the availability of large-scale annotated datasets and deep convolutional neural networks (CNNs). CNNs enable learning data-driven, highly representative, hierarchical image features from sufficient training data. However, obtaining datasets as comprehensively annotated as ImageNet in the medical imaging domain remains a challenge. There are currently three major techniques that successfully employ CNNs to medical image classification: training the CNN from scratch, using off-the-shelf pre-trained CNN features, and conducting unsupervised CNN pre-training with supervised fine-tuning. Another effective method is transfer learning, i.e., fine-tuning CNN models pre-trained from natural image dataset to medical image tasks. In this paper, we exploit three important, but previously understudied factors of employing deep convolutional neural networks to computer-aided detection problems. We first explore and evaluate different CNN architectures. The studied models contain 5 thousand to 160 million parameters, and vary in numbers of layers. We then evaluate the influence of dataset scale and spatial image context on performance. Finally, we examine when and why transfer learning from pre-trained ImageNet (via fine-tuning) can be useful. We study two specific computer-aided detection (CADe) problems, namely thoraco-abdominal lymph node (LN) detection and interstitial lung disease (ILD) classification. We achieve the state-of-the-art performance on the mediastinal LN detection, and report the first five-fold cross-validation classification results on predicting axial CT slices with ILD categories. Our extensive empirical evaluation, CNN model analysis and valuable insights can be extended to the design of high performance CAD systems for other medical imaging tasks.

Despite tremendous progress in computer vision, effective learning on very large-scale (> 100K patients) medical image databases has been vastly hindered. We present an interleaved text/image deep learning system to extract and mine the semantic interactions of radiology images and reports from a national research hospital's picture archiving and communication system. Instead of using full 3D medical volumes, we focus on a collection of representative ~216K 2D key images/slices (selected by clinicians for diagnostic reference) with text-driven scalar and vector labels. Our system interleaves between unsupervised learning (e.g., latent Dirichlet allocation, recurrent neural net language models) on document- and sentence-level texts to generate semantic labels and supervised learning via deep convolutional neural networks (CNNs) to map from images to label spaces. Disease-related key words can be predicted for radiology images in a retrieval manner. We have demonstrated promising quantitative and qualitative results. The large-scale datasets of extracted key images and their categorization, embedded vector labels and sentence descriptions can be harnessed to alleviate the deep learning “data-hungry” obstacle in the medical domain.

This special issue consists of 12 research papers. The papers cover a wide range of important topics including novel imaging, computational modeling, automatic vertebra segmentation, as well as computer assisted intervention. Among the 12 papers, nine are related to medical image analysis and three are in the scope of computer assisted intervention. The papers cover a variety of imaging modalities, include CT, MR, X-ray, ), ultrasound, and multi-modality.

Due to its importance and possible applications in visualization, tumor detection and preoperative planning, automatic small bowel segmentation is essential for computer-aided diagnosis of small bowel pathology. However, segmenting the small bowel directly on computed tomography (CT) scans is very difficult because of the low image contrast on CT scans and high tortuosity of the small bowel and its close proximity to other abdominal organs. Motivated by the intensity characteristics of abdominal CT images, the anatomic relationship between the mesenteric vasculature and the small bowel, and potential usefulness of the mesenteric vasculature for establishing the path of the small bowel, we propose a novel mesenteric vasculature map-guided method for small bowel segmentation on high-resolution CT angiography scans. The major mesenteric arteries are first segmented using a vessel tracing method based on multi-linear subspace vessel model and Bayesian inference. Second, multi-view, multi-scale vesselness enhancement filters are used to segment small vessels, and vessels directly or indirectly connecting to the superior mesenteric artery are classified as mesenteric vessels. Third, a mesenteric vasculature map is built by linking vessel bifurcation points, and the small bowel is segmented by employing the mesenteric vessel map and fuzzy connectness. The method was evaluated on 11 abdominal CT scans of patients suspected of having carcinoid tumors with manually labeled reference standard. The result, 82.5% volume overlap accuracy compared with the reference standard, shows it is feasible to segment the small bowel on CT scans using the mesenteric vasculature as a roadmap.

In this paper, we propose a novel method based on a strategic combination of the active appearance model (AAM), live wire (LW), and graph cuts (GCs) for abdominal 3-D organ segmentation. The proposed method consists of three main parts: model building, object recognition, and delineation. In the model building part, we construct the AAM and train the LW cost function and GC parameters. In the recognition part, a novel algorithm is proposed for improving the conventional AAM matching method, which effectively combines the AAM and LW methods, resulting in the oriented AAM (OAAM). A multiobject strategy is utilized to help in object initialization. We employ a pseudo-3-D initialization strategy and segment the organs slice by slice via a multiobject OAAM method. For the object delineation part, a 3-D shape-constrained GC method is proposed. The object shape generated from the initialization step is integrated into the GC cost computation, and an iterative GC-OAAM method is used for object delineation. The proposed method was tested in segmenting the liver, kidneys, and spleen on a clinical CT data set and also on the MICCAI 2007 Grand Challenge liver data set. The results show the following: 1) The overall segmentation accuracy of true positive volume fraction TPVF > 94.3% and false positive volume fraction can be achieved; 2) the initialization performance can be improved by combining the AAM and LW; 3) the multiobject strategy greatly facilitates initialization; 4) compared with the traditional 3-D AAM method, the pseudo-3-D OAAM method achieves comparable performance while running 12 times faster; and 5) the performance of the proposed method is comparable to state-of-the-art liver segmentation algorithm. The executable version of the 3-D shape-constrained GC method with a user interface can be downloaded from http://xinjianchen.wordpress.com/research/.

Cardiac CT (CCT) is widely available and has been validated for the detection of focal myocardial scar using a delayed enhancement technique in this paper. CCT, however, has not been previously evaluated for quantification of diffuse myocardial fibrosis. In our investigation, we sought to evaluate the potential of low-dose CCT for the measurement of myocardial whole heart extracellular volume (ECV) fraction. ECV is altered under conditions of increased myocardial fibrosis. A framework consisting of three main steps was proposed for CCT whole heart ECV estimation. First, a shape-constrained graph cut (GC) method was proposed for myocardium and blood pool segmentation on postcontrast image. Second, the symmetric demons deformable registration method was applied to register precontrast to postcontrast images. So the correspondences between the voxels from precontrast to postcontrast images were established. Finally, the whole heart ECV value was computed. The proposed method was tested on 20 clinical low-dose CCT datasets with precontrast and postcontrast images. The preliminary results demonstrated the feasibility and efficiency of the proposed method.

In this paper, we present an automatic renal cortex segmentation approach using the implicit shape registration and novel multiple surfaces graph search. The proposed approach is based on a hierarchy system. First, the whole kidney is roughly initialized using an implicit shape registration method, with the shapes embedded in the space of Euclidean distance functions. Second, the outer and inner surfaces of renal cortex are extracted utilizing multiple surfaces graph searching, which is extended to allow for varying sampling distances and physical constraints to better separate the renal cortex and renal column. Third, a renal cortex refining procedure is applied to detect and reduce incorrect segmentation pixels around the renal pelvis, further improving the segmentation accuracy. The method was evaluated on 17 clinical computed tomography scans using the leave-one-out strategy with five metrics: Dice similarity coefficient (DSC), volumetric overlap error (OE), signed relative volume difference (SVD), average symmetric surface distance (D(avg)), and average symmetric rms surface distance (D(rms)). The experimental results of DSC, OE, SVD, D(avg) , and D(rms) were 90.50% ± 1.19%, 4.38% ± 3.93%, 2.37% ± 1.72%, 0.14 mm ± 0.09 mm , and 0.80 mm ± 0.64 mm, respectively. The results showed the feasibility, efficiency, and robustness of the proposed method.

In the above-named article (ibid., vol. 31, no. 10, pp. 1849-1860, Oct. 2012), the author name Jian Tian should have been Jie Tian.

Traumatic injury of the pelvis is common and potentially devastating, with pelvic fractures being a major cause of trauma patient mortality. Detection and management of pelvic injuries is challenging due to varying injury patterns and resulting complications such as hemorrhage and infection. In this paper, we investigate the application of computer-aided detection (CAD) techniques for pelvic fracture detection. We propose a fast semi-automated method of pelvic fracture detection using a combination of (i) graph cuts and (ii) mean and Gaussian curvatures. A fracture is modeled as a minimum cut in a weighted graph. The same fracture is alternatively modeled as a valley based on the signs of mean and Gaussian curvatures. Each of these methods, in isolation, generates false positives in addition to the true fracture. We then combine the two methods and perform a neighborhood analysis to eliminate the false positives. Experimental results indicate that proposed method is very promising.

The ability to accurately locate a polyp found on computed tomographic colonography (CTC) at subsequent optical colonoscopy (OC) is an important part of CTC for colorectal cancer screening. A previous study has shown that a polyp's normalized distance along the colon centerline (NDACC) derived from CTC data can be utilized to predict its location at OC. We present a novel and automated method using a high degree uniform B-Spline curve fitting of the colon centerline on CTC to simulate OC colonoscope paths in order to more accurately predict polyp position. This evaluation is to determine whether the B-Spline method produces statistically significant improvement over the NDACC method in terms of prediction errors within 5 and 10 cm (± 1 colonoscope mark). Two-tailed Fisher's Exact Tests indicate the B-Spline method is superior to the NDACC method, especially for locating polyps in the central portion of the colon. The ability to predict polyp locations within 5 and 10 cm increased 31.5 % and 10.2 % respectively using the B-Spline method.

In this paper, we propose a novel 3D automatic anatomy segmentation method based on the synergistic combination of active appearance models (AAM), live wire (LW) and graph cut (GC). The proposed method consists of three main parts: model building, initialization and segmentation. For the model building part, an AAM model is constructed and the LW cost function is trained. For the initialization part, an improved iterative model refinement algorithm is proposed for the AAM optimization, which synergistically combines the AAM and LW method (OAAM). And a multi-object strategy is applied to help the object initialization. A pseudo 3D initialization strategy is employed to segment the organs slice by slice via multi-object OAAM method. The model constraints are applied to the initialization result. For the segmentation part, the object shape information generated from the initialization step is integrated into the GC cost computation. And an iterative GCOAAM method is proposed for object delineation. This method is a general method and can be applied to any organ segmentation. The proposed method was tested on the clinical liver and kidney CT data sets. The results showed the following: (a) an overall segmentation accuracy of true positive fraction>93.5%, and false positive fraction<0.2% can be achieved. (b) The initialization performance is improved by combining the AAM and LW. (c) The multi-object strategy greatly helps the initialization due to inter-object constraints.

Monitoring the evolution of renal cancer is essential in clinical oncology trials. However, manual tumor measurements are subjective and inconsistent. We have developed and implemented an efficient image-morphing-based method to track dimensional measurements in serial medical images. Our method extracts feature blocks centered at edges detected in source and target images, and then applies a fast, approximate nearest-neighbor algorithm to match these features. This matching gives rise to a global feature flow, and this feature flow can be used to track dimensional measurements.

Colon cancer is the second leading cause of cancer-related deaths in the United States. Computed tomographic colonography (CTC) combined with a computer aided detection system provides a feasible approach for improving colonic polyps detection and increasing the use of CTC for colon cancer screening. To distinguish true polyps from false positives, various features extracted from polyp candidates have been proposed. Most of these traditional features try to capture the shape information of polyp candidates or neighborhood knowledge about the surrounding structures (fold, colon wall, etc.). In this paper, we propose a new set of shape descriptors for polyp candidates based on statistical curvature information. These features called histograms of curvature features are rotation, translation and scale invariant and can be treated as complementing existing feature set. Then in order to make full use of the traditional geometric features (defined as group A) and the new statistical features (group B) which are highly heterogeneous, we employed a multiple kernel learning method based on semi-definite programming to learn an optimized classification kernel from the two groups of features. We conducted leave-one-patient-out test on a CTC dataset which contained scans from 66 patients. Experimental results show that a support vector machine (SVM) based on the combined feature set and the semi-definite optimization kernel achieved higher FROC performance compared to SVMs using the two groups of features separately. At a false positive per scan rate of 5, the sensitivity of the SVM using the combined features improved from 0.77 (Group A) and 0.73 (Group B) to 0.83 (p ≤ 0.01).

Low noise, high resolution, fast and accurate T₁ maps from MRI images of the brain can be performed using a dual flip angle method. However, B₁ field inhomogeneity, which is particularly problematic at high field strengths (e.g., 3T), limits the ability of the scanner to deliver the prescribed flip angle, introducing errors into the T₁ maps that limit the accuracy of quantitative analyses based on those maps. A dual repetition time method was used for acquiring a B₁ map to correct that inhomogeneity. Additional inaccuracies due to misregistration of the acquired T₁-weighted images were corrected by rigid registration, and the effects of misalignment on the T₁ maps were compared to those of B₁ inhomogeneity in 19 normal subjects. However, since B₁ map acquisition takes up precious scanning time and most retrospective studies do not have B₁ map, we designed a template-based correction strategy. B₁ maps from different subjects were aligned using a twelve-parameter affine registration. Recomputed T₁ maps showed an important improvement with respect to the noncorrected maps: histograms of all corrected maps exhibited two peaks corresponding to white and gray matter tissues, while unimodal histograms were observed in all uncorrected maps because of the inhomogeneity. A method to detect the best nonsubject-specific B₁ correction based on a set of features was designed. The optimum set of weighting factors for those features was computed. The best available B₁ correction was detected in almost all subjects while corrections comparable to the T₁ map corrected using the B₁ map from the same subject were detected in the others.

Colon cancer is the second leading cause of cancer-related deaths per year in industrial nations. Virtual colonoscopy is a new, less invasive alternative to the usually practiced optical colonoscopy for colorectal polyp and cancer screening. In this paper, we present some physics-based modeling and pattern recognition techniques to identify anatomical landmarks in the human colon like the haustral folds and the tenia coli to further exploit the benefits of virtual colonoscopy. A combination of heat diffusion field algorithm and fuzzy c-means clustering algorithm is used to detect the haustral folds in human colon from volumetric computed tomography (CT) images. Each voxel on the corresponding colon surface is parameterized using the colon centerline information and associated local Frenet frames. The parameterized fold information is utilized to establish the tentative location of one tenia coli. Preliminary results on automated detection of tenia coli are shown on the colon surface.

Computed tomographic colonography (CTC) provides a feasible way for the detection of colorectal polyps and cancer screening. In the clinical practice of CTC, a true colonic polyp will be confirmed with high confidence if a radiologist can find it in both the supine and prone scans. To assist radiologists in CTC reading, we propose a new colonic polyp matching method based on statistical curvature information of polyp candidates. We first extract histograms of curvature-related features (HCF) from each polyp candidate, then use diffusion map to embed the original high dimensional data into a low-dimensional space. Experimental results show that by using our HCF method, we can improve the sensitivity from 0.58 to 0.74 at false positive rate 0.1 compared with a traditional method that uses only means of curvature-related features.

Lymphangioleiomyomatosis (LAM) is a multisystem disorder associated with proliferation of smooth muscle-like cells, which leads to destruction of lung parenchyma. Subjective grading of LAM on HRCT is imprecise and can be arduous especially in cases with severe involvement. We propose a computer-aided evaluation system that grades LAM involvement based on analysis of lung texture patterns. A committee of support vector machines is employed for classification. The system was tested on 36 patients. The computer grade demonstrates good correlation with subjective radiologist grade (R=0.91, p<0.0001) and pulmonary functional tests (R=0.85, p<0.0001). The grade also provides precise progression assessment of disease over time.

Ankylosing Spondylitis is a disease characterized by abnormal bone structures (syndesmophytes) growing at intervertebral disk spaces. Because this growth is so slow as to be undetectable on plain radiographs taken over years, it is desirable to resort to computerized techniques to complement qualitative human judgment with precise quantitative measures. We developed an algorithm with minimal user intervention that provides such measures using high-resolution computed tomography (CT) images. To the best of our knowledge it is the first time that determination of the disease's status is attempted by direct measurement of the syndesmophytes. The first part of our algorithm segments the whole vertebral body using a 3-D multiscale cascade of successive level sets. The second part extracts the continuous ridgeline of the vertebral body where syndesmophytes are located. For that we designed a novel level set implementation capable of evolving on the isosurface of an object represented by a triangular mesh using curvature features. The third part of the algorithm segments the syndesmophytes from the vertebral body using local cutting planes and quantitates them. We present experimental work done with 10 patients from each of which we processed five vertebrae. The results of our algorithm were validated by comparison with a semi-quantitative evaluation made by a medical expert who visually inspected the CT scans. Correlation between the two evaluations was found to be 0.936 ( p < 10(-18)) .

CT colonography (CTC) is an emerging noninvasive technique for screening and diagnosing colon cancers. Computer aided detection (CAD) techniques can increase sensitivity and reduce false positives. We propose to employ topographical height maps in our CAD pipeline. For every detection, a height map is computed using a ray-casting algorithm. Since colonic polyps are protrusions outward from the colon wall and are round in contour, their height maps present concentric patterns. The projection direction is optimized through a multi-scale spherical search. We derive several topographic features from the map, and also compute texture features from the Haar wavelet coefficients. We send the selected features to a committee of support vector machines for classification. We have tested our method on 1186 patients with 226 polyps. Results showed that the height map features can reduce false positives by about 50%.

This paper presents an automatic method to segment and correct motion artifact on dynamic contrast enhanced magnetic resonance imaging (DCE-MRI). The breast region is segmented from DCE-MRI using mathematical morphology, region growing, and active contour models. The motion artifact presented in the image is then corrected by applying B-spline curve fitting, active contour model and forward mapping algorithm. Our segmentation method has been tested on 72 DCR-MRI studies from 33 patients. The average segmentation accuracy was 96.76%, and the confidence interval was [95.55%, 97.97%] with p<0.05. Simulation and validation experiments for motion correction are working in progress. The detail experimental results were presented at the conference. The paper represents work-in-progress in our effort to build a CAD system for breast DCE-MRI

An automatic method to segment colonic polyps in computed tomography (CT) colonography is presented in this paper. The method is based on a combination of knowledge-guided intensity adjustment, fuzzy c-mean clustering, and deformable models. The computer segmentations were compared with manual segmentations to validate the accuracy of our method. An average 76.3% volume overlap percentage among 105 polyp detections was reported in the validation, which was very good considering the small polyp size. Several experiments were performed to investigate the intraoperator and interoperator repeatability of manual colonic polyp segmentation. The investigation demonstrated that the computer-human repeatability was as good as the interoperator repeatability. The polyp segmentation was also applied in computer-aided detection (CAD) to reduce the number of false positive (FP) detections and provide volumetric features for polyp classification. Our segmentation method was able to eliminate 30% of FP detections. The volumetric features computed from the segmentation can further reduce FP detections by 50% at 80% sensitivity.

Deformable 2D-3D medical image registration is an essential technique in computer integrated surgery (CIS) to fuse 3D pre-operative data with 2D intra-operative data. Several factors may affect the accuracy of 2D-3D registration, including the number of 2D views, the angle between views, the view angle relative to anatomical objects, the co-registration error between views, the image noise, and the image distortion. In this paper, we investigate and assess the relationship between these factors and the accuracy of 2D-3D registration. We proposed a deformable 2D-3D registration method based on a statistical model. We conducted experiments using a hemi-pelvis model and simulated X-ray images. Some discussions are provided on how to improve the accuracy of 2D-3D registration based on our assessment.

In this paper we present a novel technique for non-rigid medical image registration and correspondence finding based on a multiple-layer flexible mesh template matching technique. A statistical anatomical model is built in the form of a tetrahedral mesh, which incorporates both shape and density properties of the anatomical structure. After the affine transformation and global deformation of the model are computed by optimizing an energy function, a multiple-layer flexible mesh template matching is applied to find the vertex correspondence and achieve local deformation. The multiple-layer structure of the template can be used to describe different scale of anatomical features; furthermore, the template matching is flexible which makes the correspondence finding robust. A leave-one-out validation has been conducted to demonstrate the effectiveness and accuracy of our method.

This paper describes an ongoing project to develop a computer-integrated system to assist surgeons in revision total hip replacement (RTHR) surgery. In RTHR surgery, a failing orthopedic hip implant, typically cemented, is replaced with a new one by removing the old implant, removing the cement and fitting a new implant into an enlarged canal broached in the femur. RTHR surgery is a difficult procedure fraught with technical challenges and a high incidence of complications. The goals of the computer-based system are the significant reduction of cement removal labor and time, the elimination of cortical wall penetration and femur fracture, the improved positioning and fit of the new implant resulting from precise, high-quality canal milling and the reduction of bone sacrificed to fit the new implant. Our starting points are the ROBODOC system for primary hip replacement surgery and the manual RTHR surgical protocol. We first discuss the main difficulties of computer-integrated RTHR surgery and identify key issues and possible solutions. We then describe possible system architectures and protocols for preoperative planning and intraoperative execution. We present a summary of methods and preliminary results in CT image metal artifact removal, interactive cement cut-volume definition and cement machining, anatomy-based registration using fluoroscopic X-ray images and clinical trials using an extended RTHR version of ROBODOC. We conclude with a summary of lessons learned and a discussion of current and future work.