Artificial intelligence (AI) shows great potential in assisting radiologists to improve the efficiency and accuracy of medical image interpretation and diagnosis. However, a versatile AI model requires large-scale data and comprehensive annotations, which are often impractical in medical settings. Recent studies leverage radiology reports as a naturally high-quality supervision for medical images, using contrastive language-image pre-training (CLIP) to develop language-informed models for radiological image interpretation. Nonetheless, these approaches typically contrast entire images with reports, neglecting the local associations between imaging regions and report sentences, which may undermine model performance and interoperability. In this paper, we propose a fine-grained vision-language model (fVLM) for anatomy-level CT image interpretation. Specifically, we explicitly match anatomical regions of CT images with corresponding descriptions in radiology reports and perform contrastive pre-training for each anatomy individually. Fine-grained alignment, however, faces considerable false-negative challenges, mainly from the abundance of anatomy-level healthy samples and similarly diseased abnormalities, leading to ambiguous patient-level pairings. To tackle this issue, we propose identifying false negatives of both normal and abnormal samples and calibrating contrastive learning from patient-level to disease-aware pairing. We curated the largest CT dataset to date, comprising imaging and report data from 69,086 patients, and conducted a comprehensive evaluation of 54 major and important disease (including several most deadly cancers) diagnosis tasks across 15 main anatomies. Experimental results demonstrate the substantial potential of fVLM in versatile medical image interpretation. In the zero-shot classification task, we achieved an average AUC of 81.3% on 54 diagnosis tasks, surpassing CLIP and supervised methods by 12.9% and 8.0%, respectively. Additionally, on the publicly available CT-RATE and Rad-ChestCT benchmarks, our fVLM outperformed the current state-of-the-art methods with absolute AUC gains of 7.4% and 4.8%, respectively.

The pancreas plays an important role in glucose metabolism, and developing diabetes or long-term glucose metabolism disturbance may be a prevalent sequela after pancreatectomy. Nevertheless, relative factors of new-onset diabetes after pancreatectomy stay unclear. Radiomics analysis is potential to identify image markers for disease prediction or prognosis. Meanwhile, combination of imaging and electronic medical record (EMR) showed superior performance than imaging or EMR alone in previous studies. One critical step is to identity predictors from high-dimensional features, and it is even more challenging to select and fuse imaging and EMR features. In this work, we develop a radiomics pipeline to assess postoperative new-onset diabetes risk of patients undergoing distal pancreatectomy. Specifically, we extract multiscale image features with 3D wavelet transformation, and include patients' characteristics, body composition and pancreas volume information as clinical features. Then, we propose a multi-view subspace clustering guided feature selection method (MSCUFS) for the selection and fusion of image and clinical features. Finally, a prediction model is constructed with classical machine learning classifier. Experimental results on an established distal pancreatectomy cohort showed that the SVM model with combined imaging and EMR features demonstrated good discrimination, with an AUC value of 0.824, which improved the model with image features alone by 0.037 AUC. Compared with state-of-the-art feature selection methods, the proposed MSCUFS has superior performance in fusing image and clinical features.

Pancreas identification and segmentation is an essential task in the diagnosis and prognosis of pancreas disease. Although deep neural networks have been widely applied in abdominal organ segmentation, it is still challenging for small organs (e.g. pancreas) that present low contrast, highly flexible anatomical structure and relatively small region. In recent years, coarse-to-fine methods have improved pancreas segmentation accuracy by using coarse predictions in the fine stage, but only object location is utilized and rich image context is neglected. In this paper, we propose a novel distance-based saliency-aware model, namely DSD-ASPP-Net, to fully use coarse segmentation to highlight the pancreas feature and boost accuracy in the fine segmentation stage. Specifically, a DenseASPP (Dense Atrous Spatial Pyramid Pooling) model is trained to learn the pancreas location and probability map, which is then transformed into saliency map through geodesic distance-based saliency transformation. In the fine stage, saliency-aware modules that combine saliency map and image context are introduced into DenseASPP to develop the DSD-ASPP-Net. The architecture of DenseASPP brings multi-scale feature representation and achieves larger receptive field in a denser way, which overcome the difficulties brought by variable object sizes and locations. Our method was evaluated on both public NIH pancreas dataset and local hospital dataset, and achieved an average Dice-Sørensen Coefficient (DSC) value of 85.49±4.77% on the NIH dataset, outperforming former coarse-to-fine methods.

Non-used clinical information has negative implications on healthcare quality. Clinicians pay priority attention to clinical information relevant to their specialties during routine clinical practices but may be insensitive or less concerned about information showing disease risks beyond their specialties, resulting in delayed and missed diagnoses or improper management. In this study, we introduced an electronic health record (EHR)-oriented knowledge graph system to efficiently utilize non-used information buried in EHRs. EHR data were transformed into a semantic patient-centralized information model under the ontology structure of a knowledge graph. The knowledge graph then creates an EHR data trajectory and performs reasoning through semantic rules to identify important clinical findings within EHR data. A graphical reasoning pathway illustrates the reasoning footage and explains the clinical significance for clinicians to better understand the neglected information. An application study was performed to evaluate unconsidered chronic kidney disease (CKD) reminding for non-nephrology clinicians to identify important neglected information. The study covered 71,679 patients in non-nephrology departments. The system identified 2,774 patients meeting CKD diagnosis criteria and 10,377 patients requiring high attention. A follow-up study of 5,439 patients showed that 82.1% of patients who met the diagnosis criteria and 61.4% of patients requiring high attention were confirmed to be CKD positive during follow-up research. The application demonstrated that the proposed approach is feasible and effective in clinical information utilization. Additionally, it's valuable as an explainable artificial intelligence to provide interpretable recommendations for specialist physicians to understand the importance of non-used data and make comprehensive decisions.

The presence of necrosis is associated with tumor progression and patient outcomes in many cancers, but existing analyses rarely adopt quantitative methods because the manual quantification of histopathological features is too expensive. We aim to accurately identify necrotic regions on hematoxylin and eosin (HE)-stained slides and to calculate the ratio of necrosis with minimal annotations on the images. An adaptive method named Learning from Label Fuzzy Proportions (LLFP) was introduced to histopathological image analysis. Two datasets of liver cancer HE slides were collected to verify the feasibility of the method by training on the internal set using cross validation and performing validation on the external set, along with ensemble learning to improve performance. The models from cross validation performed relatively stably in identifying necrosis, with a Concordance Index of the Slide Necrosis Score (CISNS) of 0.9165±0.0089 in the internal test set. The integration model improved the CISNS to 0.9341 and achieved a CISNS of 0.8278 on the external set. There were significant differences in survival (p = 0.0060) between the three groups divided according to the calculated necrosis ratio. The proposed method can build an integration model good at distinguishing necrosis and capable of clinical assistance as an automatic tool to stratify patients with different risks or as a cluster tool for the quantification of histopathological features. We presented a method effective for identifying histopathological features and suggested that the extent of necrosis, especially micronecrosis, in liver cancer is related to patient outcomes.