Accurately measuring protein-RNA binding affinity is crucial in many biological processes and drug design. Previous computational methods for protein-RNA binding affinity prediction rely on either sequence or structure features, unable to capture the binding mechanisms comprehensively. The recent emerging pre-trained language models trained on massive unsupervised sequences of protein and RNA have shown strong representation ability for various in-domain downstream tasks, including binding site prediction. However, applying different-domain language models collaboratively for complex-level tasks remains unexplored. In this paper, we propose CoPRA to bridge pre-trained language models from different biological domains via Complex structure for Protein-RNA binding Affinity prediction. We demonstrate for the first time that cross-biological modal language models can collaborate to improve binding affinity prediction. We propose a Co-Former to combine the cross-modal sequence and structure information and a bi-scope pre-training strategy for improving Co-Former's interaction understanding. Meanwhile, we build the largest protein-RNA binding affinity dataset PRA310 for performance evaluation. We also test our model on a public dataset for mutation effect prediction. CoPRA reaches state-of-the-art performance on all the datasets. We provide extensive analyses and verify that CoPRA can (1) accurately predict the protein-RNA binding affinity; (2) understand the binding affinity change caused by mutations; and (3) benefit from scaling data and model size.

Visual grounding is essential for precise perception and reasoning in multimodal large language models (MLLMs), especially in medical imaging domains. While existing medical visual grounding benchmarks primarily focus on single-image scenarios, real-world clinical applications often involve sequential images, where accurate lesion localization across different modalities and temporal tracking of disease progression (e.g., pre- vs. post-treatment comparison) require fine-grained cross-image semantic alignment and context-aware reasoning. To remedy the underrepresentation of image sequences in existing medical visual grounding benchmarks, we propose MedSG-Bench, the first benchmark tailored for Medical Image Sequences Grounding. It comprises eight VQA-style tasks, formulated into two paradigms of the grounding tasks, including 1) Image Difference Grounding, which focuses on detecting change regions across images, and 2) Image Consistency Grounding, which emphasizes detection of consistent or shared semantics across sequential images. MedSG-Bench covers 76 public datasets, 10 medical imaging modalities, and a wide spectrum of anatomical structures and diseases, totaling 9,630 question–answer pairs. We benchmark both general-purpose MLLMs (e.g., Qwen2.5-VL) and medical-domain specialized MLLMs (e.g., HuatuoGPT-vision), observing that even the advanced models exhibit substantial limitations in medical sequential grounding tasks. To advance this field, we construct MedSG-188K, a large-scale instruction-tuning dataset tailored for sequential visual grounding, and further develop MedSeq-Grounder, an MLLM designed to facilitate future research on fine-grained understanding across medical sequential images. We release all resources on https://github.com/Yuejingkun/MedSG-Bench

Federated learning (FL) has emerged as a promising distributed paradigm that enables collaborative model training while preserving data privacy, but it suffers from performance degradation due to data heterogeneity. Although clustered federated learning (CFL) attempts to address this challenge by grouping clients with similar data distributions, existing methods are inefficient in capturing client data representations, leading to incorrect cluster identities and inferior cluster performance. To overcome these limitations, we propose an efficient prototype-based CFL framework (FedPC). Specifically, we introduce a dual-prototype strategy combining specific prototypes and generalized prototypes to capture class representations for cluster identities, along with a prototype-contrastive training mechanism that maximizes intra-cluster prototype consistency to improve cluster performance. Extensive experiments on medical imaging datasets (BloodMNIST and DermaMNIST) demonstrate that the FedPC outperforms nine state-of-the-art (SOTA) approaches, achieving average improvements of 2.17% and 3.47%, respectively. Furthermore, the FedPC reduces communication overhead by 3.33 to 5.68 times compared to SOTA methods, showcasing its efficiency in real-world FL scenarios.

Medical images are often more difficult to acquire than natural images due to the specialized equipment and technology required, leading to fewer available medical image datasets. This limitation poses challenges in training robust pretrained medical vision models. How to best leverage natural pretrained vision models and adapt them to the medical domain remains an open question. For image classification, linear probing (LP) is a commonly used technique. However, LP primarily focuses on the output after feature extraction, without addressing the inherent differences between medical images and natural image-based pretrained models. To bridge this gap, we introduce visual prompting (VP) and investigate strategies for integrating LP and VP. We propose a joint learning framework with a composite loss function that includes a categorization loss and a discrepancy loss, capturing the variance between prompted and plain images, naming this joint training strategy MoVL (Mixture of Visual Prompting and Linear Probe). We experiment on four medical image classification datasets, with two mainstream architectures, ResNet and CLIP. Results shows that without changing the parameters and architecture of backbone model and with less parameters, there is potential for MoVL to achieve full finetune (FF) accuracy (on four medical datasets, average 90.91% for MoVL and 91.13% for FF). On out of distribution medical dataset, our method (90.33%) can outperform FF (85.15%) with absolute 5.18 % lead.

Deep learning-based image compression algorithms have witnessed remarkable advancements. However, compressing high-resolution images often suffers from color bias and joint seams between reconstructed patches as it ignores the global features of whole images. To address this issue, we propose a novel image compression autoencoder based on the local-global joint attention mechanism (CAE-LG). Specifically, CAE-LG extracts local and global features, and then refines and fuses these two features to obtain the reconstructed image. We introduce a feature attention block that leverages dynamic convolution and attention layers to enhance the encoding and decoding of feature maps. We validated our method on datasets comprising both natural and medical images. Experimental results demonstrate that our proposed approach outperforms the existing image compression methods. In addition, performance improvements on different datasets show the generality of our model. Ablation studies confirm the effectiveness of both the joint learning of local and global features and the feature attention block.

Deep learning (DL) models have achieved remarkable success in various domains. But training an accurate DL model requires large amounts of data, which can be challenging to obtain in medical settings due to privacy concerns. Recently, federated learning (FL) has emerged as a promising solution that shares local models instead of raw data. However, FL in medical settings faces challenges of client drift due to the data heterogeneity across dispersed institutions. Although there exist studies to address this challenge, they mainly focus on the classification tasks that learn global representation of an entire image. Few have been studied on the dense prediction tasks, such as object detection. In this study, we propose dense contrastive-based federated learning (DCFL) tailored for dense prediction tasks in FL settings. DCFL introduces dense contrastive learning to FL, which aligns the local optimization objectives towards the global objective by maximizing the agreement of representations between the global and local models. Moreover, to improve the performance of dense target prediction at each level, DCFL applies multi-scale contrastive representation by utilizing multi-scale representations with dense features in contrastive learning. We evaluated DCFL on a set of realistic datasets for pulmonary nodule detection. DCFL demonstrates an overall performance improvement compared with the other federated learning methods in heterogeneous settings-improving the mean average precision by 4.13% and testing recall by 6.07% in highly heterogeneous settings.

Large Language models have made remarkable strides in natural language understanding and generation. However, their performance in specialized fields like medicine often falls short due to the lack of domain-specific knowledge during pre-training. While fine-tuning on labeled medical data is a common approach for task adaptation, it may not capture the comprehensive medical knowledge required. In this paper, we proposed a Hybrid Granularity Mask Learning (HGM) method for domain adaptation in the medical field. Our method incorporates multi-level linguistic characteries including token, entity, and subsentence to enable the model to acquire medical knowledge comprehensively. We fine-tune a medical-specific language model derived from ChatGLM-6B and Bloom-7B on downstream medical tasks and evaluate its performance. The results demonstrate a significant improvement compared to the baseline, thus affirming the effectiveness of our proposed method.

Federated learning (FL) enables collaborative model training across clients while preserving data privacy. However, FL faces the challenge of data heterogeneity, leading to biased local models that deviate from the global model during optimization. And existing FL algorithms like Federated Averaging (FedAvg) suffer from this issue. To address this problem, we propose a novel approach called multi-branch prototype federated learning (FedMBP). FedMBP creates auxiliary branches within each local model to integrate different levels of local and global prototypes, thus preventing local model drift by aligning local prototypes with global ones. We also introduce mixed cross-entropy on the auxiliary branches to effectively transfer global prototype knowledge to local models. We conduct experiments on three publicly available datasets, including natural and medical image domains. Our experiments demonstrate that FedMBP outperforms existing FL algorithms, achieving superior model performance.

Previous studies showed that facial appearance is an important phenotypic indicator of human diseases or biological conditions. Recent advancements in deep learning have shown great potential in facial image analysis, including health status assessments. However, prior methods mainly focused on single modality analysis of either the 2D texture images or 3D facial meshes, which are limited in their ability to fully capture the relationships between biometric measurements and diseases. To address these issues, we propose a task-adaptive multi-modal fusion network, TAMM, for facial-related health assessments. Our model leverages both the geometric and texture features of 3D facial images by a task-adaptive Transformer (TAFormer), which can dynamically extract features from different modalities and scales for various tasks via spatial attention and cross modal multi-scale attention, effectively capture intra- and inter-modal relationships between features. Experimental results on a dataset of 19,775 patients demonstrate that TAMM achieves the state-of-the-art performance on various regression and classification tasks including age, BMI, and fatty liver disease predictions. Ablation studies shows the importance of multi-modal fusion and task-specific adaptability of our model in achieving optimal performance.

Automated radiology report generation aims at automatically generating a detailed description of medical images, which can greatly alleviate the workload of radiologists and provide better medical services to remote areas. Most existing works pay attention to the holistic impression of medical images, failing to utilize important anatomy information. However, in actual clinical practice, radiologists usually locate important anatomical structures, and then look for signs of abnormalities in certain structures and reason the underlying disease. In this paper, we propose a novel framework AGFNet to dynamically fuse the global and anatomy region feature to generate multi-grained radiology report. Firstly, we extract important anatomy region features and global features of input Chest X-ray (CXR). Then, with the region features and the global features as input, our proposed self-adaptive fusion gate module could dynamically fuse multi-granularity information. Finally, the captioning generator generates the radiology reports through multi-granularity features. Experiment results illustrate that our model achieved the state-of-the-art performance on two benchmark datasets including the IU X-Ray and MIMIC-CXR. Further analyses also prove that our model is able to leverage the multi-grained information from radiology images and texts so as to help generate more accurate reports.

Recent developments of deep learning methods have demonstrated their feasibility in liver malignancy diagnosis using ultrasound (US) images. However, most of these methods require manual selection and annotation of US images by radiologists, which limit their practical application. On the other hand, US videos provide more comprehensive morphological information about liver masses and their relationships with surrounding structures than US images, potentially leading to a more accurate diagnosis. Here, we developed a fully automated artificial intelligence (AI) pipeline to imitate the workflow of radiologists for detecting liver masses and diagnosing liver malignancy. In this pipeline, we designed an automated mass-guided strategy that used segmentation information to direct diagnostic models to focus on liver masses, thus increasing diagnostic accuracy. The diagnostic models based on US videos utilized bi-directional convolutional long short-term memory modules with an attention-boosted module to learn and fuse spatiotemporal information from consecutive video frames. Using a large-scale dataset of 50 063 US images and video frames from 11 468 patients, we developed and tested the AI pipeline and investigated its applications. A dataset of annotated US images is available at https://doi.org/10.5281/zenodo.7272660.

Artificial Intelligence (AI) based diagnosis systems have emerged as powerful tools to reform traditional medical care. Each clinician now wants to have his own intelligent diagnostic partner to expand the range of services he can provide. However, the implementation of intelligent decision support systems based on clinical note has been hindered by the lack of extensibility of end-to-end AI diagnosis algorithms. When reading a clinical note, expert clinicians make inferences with relevant medical knowledge, which serve as prompts for making accurate diagnoses. Therefore, external medical knowledge is commonly employed as an augmentation for medical text classification tasks. Existing methods, however, cannot integrate knowledge from various knowledge sources as prompts nor can fully utilize explicit and implicit knowledge. To address these issues, we propose a Medical Knowledge-enhanced Prompt Learning (MedKPL) diagnostic framework for transferable clinical note classification. Firstly, to overcome the heterogeneity of knowledge sources, such as knowledge graphs or medical QA databases, MedKPL uniform the knowledge relevant to the disease into text sequences of fixed format. Then, MedKPL integrates medical knowledge into the prompt designed for context representation. Therefore, MedKPL can integrate knowledge into the models to enhance diagnostic performance and effectively transfer to new diseases by using relevant disease knowledge. The results of our experiments on two medical datasets demonstrate that our method yields superior medical text classification results and performs better in cross-departmental transfer tasks under few-shot or even zero-shot settings. These findings demonstrate that our MedKPL framework has the potential to improve the interpretability and transferability of current diagnostic systems.

Assertion information is of crucial importance for constructing an intelligent diagnosis system as it contains clinical findings and decision basis of clinicians in the electronic medical records (EMRs), e.g., whether a symptom is present or not. Current work mainly treats assertion detection as a sequence labeling task, or constructs rule-based methods. However, there is a challenge that to detect assertions embedded in the context with long-range dependencies, needs considering the whole text to capture the complex linguistic and underlying semantic information. To tackle the above issues, we consider assertion detection as a semantic reasoning task based on natural language inference (NLI). First, we generate candidate spans with boundary detection on the basis of which we can enrich the training corpus with external knowledge such as assertion definitions. Then we detect the assertions through the NLI-based classification. To the best of our knowledge, we build the first Chinese assertion dataset, which contains 4237 sentences on privacy de-identified ophthalmology remote reading reports. Extensive experiments demonstrate that our proposed method achieves the best results on both of the English dataset i2b2 and the Chinese privacy dataset.

Recent studies have found that human faces not only encode personal identity information, but also related to plenty of human health information. However, for health status prediction, bias remains a challenge which may be introduced by the intrinsic dependency between variables. For example, demographic variables such as gender or age are associated to many diseases, causing model learn to infer the age or gender rather than the real pathological cues. In order to eliminate the bias and guarantee the high performance of health prediction from facial images, we propose a novel unbiased model based on attention mechanism and adversarial technique, named AMAT-Net for health prediction. More specifically, we minimize the target prediction loss and maximize the bias variable prediction loss during training, to encourage extracted features to keep predictive ability while decouple with the bias. At the same time, the tansformer module is introduced to adaptively extract the features of different tasks. We constructed a dataset composed of facial images with demographic information and disease labels. The experimental results demonstrate that our model not only shows superior performance on metabolic disease prediction but also largely reduce the prediction bias on different age groups.

Computed tomography (CT) is one of the most imaging methods widely used to locate lesions such as nodules, tumors, and cysts, and make primary diagnosis. For clearer imaging of anatomical or lesions, contrast-enhanced CT (CECT) scans are imaging with injecting a contrast agent into a patient during examination. But there are limits to iodine contrast injections so that CECT scans are not convenient like non-contrast enhanced CT (NECT). Recently, deep learning models bring impressive results in computer vision, including image translation. So, we would like to apply image translation methods to generate CECT images from the more accessible NECT images, and evaluate the effects of generated images on image detection tasks. In this study, we propose a method called cross-modal enhancement training strategy for thyroid anatomy detection, which employs CycleGAN to translate non-constrast enhanced CT images to enhanced CT style images with content reserved. The experiments are conducted on thyroid CT images with anatomy object annotation. The experimental results show that by adding translated images into the training dataset, the performance of thyroid anatomy detection can be effectively improved. We achieve the best mAP of 82.5% compared to 73.2% in the along non-contrast enhanced CT training.

Lung cancer is one of the leading causes of death worldwide. Early diagnosis through cancer screening can significantly improve lung cancer patients’ survival. Recently, deep learning based diagnostic systems for nodule detection have shown great potential in assisting radiologists to screen cancer more efficiently. However, studies have found that deep learning models lack robustness against imperceptible crafted adversarial attacks and few studied improving the robustness of pulmonary nodule detection. Therefore, making pulmonary nodule detection models robust remains challenges. Moreover, traditional adversarial training methods either hurt the natural generalization or need expensive computational cost. To address these challenges, here we propose a novel adversarial training method called, Adaptive Iteration Adversarial Training (AIAT). AIAT generates adversarial samples by adding adversarial noise with an adaptive iteration strategy, so that it can stably and fast train models with improving robustness. Extensive experiments on the LUNA 16 dataset show that AIAT improves robustness for pulmonary nodule detection without compromising the natural generalization, and largely reduces training time.

Object detection, a crucial component of medical image analysis, provides physicians with an interpretable auxiliary diagnostic basis. Although existing object detection models have had great success with natural images, the growing resolution of medical images makes the problem especially challenging because of the increased expectations to exploit the image details and discover small targets in images. For instance, lesions are occasionally diminutive relative to high-resolution medical images. To address this problem, we present YOLO-SG, a salience-guided (SG) deep learning model that improves small object detection by attending to detailed regions via a generated salience map. YOLO-SG performs two rounds of detection: coarse detection and salience-guided detection. In the first round of coarse detection, YOLO-SG detects objects using a deep convolutional detection model and proposes a salience map utilizing the context surrounding objects to guide the subsequent round of detection. In the second round, YOLO-SG extracts salient regions from the original input image based on the generated salience map and combines local detail with global context information to improve the object detection performance. The experimental results demonstrate that YOLO-SG outperforms the state-of-the-art models, especially when detecting small objects.

Central precocious puberty (CPP) is the most common type of precocious puberty and has a significant effect on children. A gonadotropin-releasing hormone (GnRH)-stimulation test is the gold standard for confirming CPP. This test, however, is costly and unpleasant for patients. Therefore, it is critical to developing alternative methods for CPP diagnosis in order to alleviate patient suffering. This study aims to develop an artificial intelligence (AI) diagnostic system for predicting response to the GnRH-stimulation test using data from laboratory tests, electronic health records (EHRs), and pelvic ultrasonography and left-hand radiography reports. The challenges are in integrating these multimodal features into a comprehensive deep learning model in order to achieve an accurate diagnosis while also accounting for the missing or incomplete modalities. To begin, we developed a dynamic multimodal variational autoencoder (DMVAE) that can exploit intrinsic correlations between different modalities to impute features for missing modalities. Next, we combined features from all modalities to predict the outcome of a CPP diagnosis. The experimental results (AUROC 0.9086) demonstrate that our DMVAE model is superior to standard methods. Additionally, we showed that by setting appropriate operating thresholds, clinicians could diagnose about two-thirds of patients with confidence (1.0 specificity). Only about one-third of patients require confirmation of their diagnoses using GnRH (or GnRH analog)-stimulation tests. To interpret the results, we implemented an explainer Shapley additive explanation (SHAP) to analyze the local and global feature attributions.

During the ongoing COVID-19 outbreak, it is critical to assess patients’ disease progression with COVID-19 pneumonia by computed tomography (CT). As most of the works focused on ground-glass opacity and consolidation segmentation of COVID-19 on CT images, lung fibrosis is relatively undervalued and less studied. Automatic segmentation and accurate measurement of lung fibrosis can potentially aid treatment planning for patients of post-COVID-19 pneumonia. However, the lack of sufficient training data hinders the fibrosis segmentation of CT images. Also, redundancy among CT images can reduce annotating efficiency. To address these issues, we propose deep active learning (AL) framework, which consists of a segmentation model called UNet-RGD, and a novel acquisition method named DeepRISS. The segmentation model consists of improved structures of residual blocks, channel gates, and dropout layers. The deep learning-based acquisition method combines uncertainty estimation and clustering for selecting representative and informative samples. Experimental results show that the AL framework can achieve state-of-the-art performance and effectively reduce the number of selected samples, saving the annotation cost by 25% to 44% compared to the non-selective approach.

Bone age assessment is critical to diagnosis of various growth disorders in children, such as endocrine, nutritional disorders and dysplasia. X-rays of hand and wrist are the most common modality used to calculate bone age. In this paper, we propose a novel approach called DeepTW3 for automatic bone age assessment from X-ray images. DeepTW3 integrates Convolutional Neural Networks (CNNs) with expertise knowledge of TW3(Tanner-Whitehouse 3nd edition)-RUS(radius, ulna and short bones) bone age assessment system. The proposed method is tested on a dataset containing 1,100 hand bone X-ray images, all of which were manually annotated with selected region of interests(ROIs). Our method achieved mean absolute errors (MAE) of 0.2685, outperforming all state-of-the-art methods. For the task of grading skeletal maturity, our method using continuous stage distribution is complementary to using the clinical TW3-RUS categorical stages when interpreting critical cases of intermediate bone stage.

As the first pass for emergency patients, triage is the most important factor affecting emergency department (ED) overcrowding. So it is crucial to develop a data-driven and evidence-based triage method to quickly identify acute and severe patients, and prevent the limited emergency resources from over-diagnosis. To address these challenges, we propose an attention based deep learning framework, named DeepTriager. Trained and tested on 70,918 clinical records, DeepTriager achieved highly accurate performance on assessment of acuity level I (endangered patients), with AUC of 0.98, which was 0.16 higher than the clinical scale method MEWS and NEWS, and 0.04 higher than traditional machine learning methods. In summary, we presented a new approach for clinical evidence based discovery using a cohort of Electronic Health Records (EHRs). This approach not only outperforms the traditional word segmentation methods but also provides evidence for interpreting the results.