As one of the primary causes of visual impairment, Diabetic Retinopathy (DR) requires accurate and robust grading to facilitate timely diagnosis and intervention. Different from conventional DR grading methods that utilize single-view images, recent clinical studies have revealed that multi-view fundus images can significantly enhance DR grading performance by expanding the field of view (FOV). However, there is a long-tailed distribution problem in fundus image analysis, i.e., a high prevalence of mild DR grades and a low prevalence of rare ones (e.g., cases of high severity), which presents a significant challenge to developing a unified model capable of detecting rare or unseen DR grades not encountered during training. In this paper, we propose ProME-DR, a Prompt-driven zero-shot DR grading framework, which leverages prompt Matching and Emulating to recognize the unseen DR categories and views beyond the training set. ProME-DR disentangles the training process into two stages to learn generalized knowledge for novel DR disease grading. Initially, ProME-DR leverages two sets of prompt units to capture semantic and inter-view consistency knowledge via a split-and-mask manner, gathering instance-level DR visual clues. Subsequently, it constructs a concept-aware emulator to generate context prompt units, linking extensible knowledge learned from the previously seen DR attributes for zero-shot DR grading. Extensive experiments conducted on eight datasets and various scenarios confirm the superiority of ProME-DR.

Abstract With the refinement of 3D imaging modality, cryo-electron tomography (cryo-ET) has emerged as a powerful technique for the structural analysis of macromolecular complexes at near-atomic resolution. Recent advancements in volumetric segmentation methods applied to cryo-ET datasets have garnered significant attention within the biomedical sector. However, existing methods rely heavily on manually labeled data, which demands highly specialized expertise, making fully supervised approaches less feasible for cryo-ET images. To address this, a number of unsupervised domain adaptation (UDA) techniques have been developed to improve segmentation network performance using unlabeled data. Nevertheless, directly applying these methods to cryo-ET image segmentation presents two major challenges: 1) the source dataset, usually obtained through simulation, contains a fixed level of noise, while the target dataset, being directly collected from raw-data from the real-world scenario, has unpredictable noise levels; 2) the source data used for training typically consists of known macromolecules, in contrast, the target domain data are often unknown, causing the model to be biased towards those known macromolecules, leading to a domain shift problem. To address such challenges, in this paper, we introduce a voxel-wise unsupervised domain adaptation approach, termed Vox-UDA, specifically for cryo-ET subtomogram segmentation. Vox-UDA incorporates a noise generation module to simulate target-like noises in the source dataset for cross-noise level adaptation, and a denoised pseudo-labeling strategy based on the improved bilateral filter to alleviate the domain shift problem. Additionally, we further consider a scenario that is more in line with the real world, where the target (experimental) dataset might not be accessible during training or has a very small sample size, and we present a voxel-wise zero-shot domain adaptation (ZSDA) approach, named Vox-ZSDA. In Vox-ZSDA, we introduce a self-supervised graph learning strategy to eliminate any dependency on the target data, accompanied by a dynamic graph contrastive learning technique to enhance the model’s sensitivity to macromolecular structures to boost the segmentation performance. More importantly, we construct the first UDA and ZSDA cryo-ET subtomogram segmentation benchmark on three experimental datasets. Extensive experimental results on multiple benchmarks and newly curated real-world datasets demonstrate the superiority of our proposed approach compared to state-of-the-art UDA and ZSDA methods.

Prompt learning has emerged as an efficient tuning paradigm for fine-tuning powerful pre-trained models on downstream tasks in specific domains. Existing efforts mainly focus on dataset-level implicit embeddings by introducing extra learnable parameters instead of fully fine-tuning large-scale visual models. However, we find that these static post-training prompts are not flexible enough to adapt various input instances within the same dataset, which might lead to the loss of the model's generalization capability. To leverage the meaningful contextual information of each input instance, in this paper, we propose a straightforward yet effective method, termed CoMoDP, to enhance visual prompt learning withContext-ModulatedDiffusionPrompts. Specifically, CoMoDP is a dual-visual prompting scheme that comprises two key components:(i) a unified visual prompt designer, producing dataset-level implicit embedding as unified prompts for efficient adaptation without corrupting the underlying information of the original image; and(ii) a diffusion prompt simulator, leveraging diffusion model's meticulous understanding of semantic structure and texture edges in the images to dynamically generate instance-level implicit embedding as diffusion prompts for input samples. Moreover, to reduce the overfitting of prompts, we also introduce momentum alignment, a self-regulating strategy that restricts the optimization region of prompts in both feature and logit spaces. Extensive experiments on various standard and few-shot datasets demonstrate that our method brings substantial improvements and yields strong domain generalization performance, compared to the state-of-the-art methods. We also demonstrate both zero-shot and out-of-distribution performance to establish the utility of our dual-visual prompting scheme CoMoDP and the efficiency of each component, without involving excessive parameters.

Cryo-Electron Tomography (cryo-ET) is a 3D imaging technology that facilitates the study of macromolecular structures at near-atomic resolution. Recent volumetric segmentation approaches on cryo-ET images have drawn widespread interest in the biological sector. However, existing methods heavily rely on manually labeled data, which requires highly professional skills, thereby hindering the adoption of fully-supervised approaches for cryo-ET images. Some unsupervised domain adaptation (UDA) approaches have been designed to enhance the segmentation network performance using unlabeled data. However, applying these methods directly to cryo-ET image segmentation tasks remains challenging due to two main issues: 1) the source dataset, usually obtained through simulation, contains a fixed level of noise, while the target dataset, directly collected from raw-data from the real-world scenario, have unpredictable noise levels. 2) the source data used for training typically consists of known macromoleculars. In contrast, the target domain data are often unknown, causing the model to be biased towards those known macromolecules, leading to a domain shift problem. To address such challenges, in this work, we introduce a voxel-wise unsupervised domain adaptation approach, termed Vox-UDA, specifically for cryo-ET subtomogram segmentation. Vox-UDA incorporates a noise generation module to simulate target-like noises in the source dataset for cross-noise level adaptation. Additionally, we propose a denoised pseudo-labeling strategy based on the improved Bilateral Filter to alleviate the domain shift problem. More importantly, we construct the first UDA cryo-ET subtomogram segmentation benchmark on three experimental datasets. Extensive experimental results on multiple benchmarks and newly curated real-world datasets demonstrate the superiority of our proposed approach compared to state-of-the-art UDA methods.

Federated learning aims at training models collaboratively across participants while protecting privacy. However, one major challenge for this paradigm is the data heterogeneity issue, where biased data preferences across multiple clients, harming the model's convergence and performance. In this paper, we first introduce powerful diffusion models into the federated learning paradigm and show that diffusion representations are effective steers during federated training. To explore the possibility of using diffusion representations in handling data heterogeneity, we propose a novel diffusion-inspired Federated paradigm with Diffusion Representation Collaboration, termed FedDifRC, leveraging meaningful guidance of diffusion models to mitigate data heterogeneity. The key idea is to construct text-driven diffusion contrasting and noise-driven diffusion regularization, aiming to provide abundant class-related semantic information and consistent convergence signals. On the one hand, we exploit the conditional feedback from the diffusion model for different text prompts to build a text-driven contrastive learning strategy. On the other hand, we introduce a noise-driven consistency regularization to align local instances with diffusion denoising representations, constraining the optimization region in the feature space. In addition, FedDifRC can be extended to a self-supervised scheme without relying on any labeled data. We also provide a theoretical analysis for FedDifRC to ensure convergence under non-convex objectives. The experiments on different scenarios validate the effectiveness of FedDifRC and the efficiency of crucial components.

With the advancement of edge computing, federated learning (FL) displays a bright promise as a privacy-preserving collaborative learning paradigm. However, one major challenge for FL is the data heterogeneity issue, which refers to the biased labeling preferences among multiple clients, negatively impacting convergence and model performance. Most previous FL methods attempt to tackle the data heterogeneity issue locally or globally, neglecting underlying class-wise structure information contained in each client. In this paper, we first study how data heterogeneity affects the divergence of the model and decompose it into local, global, and sampling drift sub-problems. To explore the potential of using intra-client class-wise structural knowledge in handling these drifts, we thus propose Federated Learning with Structural Knowledge Collaboration (FedSKC). The key idea of FedSKC is to extract and transfer domain preferences from inter-client data distributions, offering diverse class-relevant knowledge and a fair convergent signal. FedSKC comprises three components: i) local contrastive learning, to prevent weight divergence resulting from local training; ii) global discrepancy aggregation, which addresses the parameter deviation between the server and clients; iii) global period review, correcting for the sampling drift introduced by the server randomly selecting devices. We have theoretically analyzed FedSKC under non-convex objectives and empirically validated its superiority through extensive experimental results. Our code is at https://github.com/hwang52/FedSKC.

Federated learning (FL) aims to train models collaboratively across clients without sharing data for privacy-preserving.However, one major challenge is the data heterogeneity issue, which refers to the biased labeling preferences at multiple clients.A number of existing FL methods attempt to tackle data heterogeneity locally (e.g., regularizing local models) or globally (e.g., fine-tuning global model), often neglecting inherent semantic information contained in each client.To explore the possibility of using intra-client semantically meaningful knowledge in handling data heterogeneity, in this paper, we propose Federated Learning with Semantic-Aware Collaboration (FedSC) to capture client-specific and class-relevant knowledge across heterogeneous clients.The core idea of FedSC is to construct relational prototypes and consistent prototypes at semantic-level, aiming to provide fruitful class underlying knowledge and stable convergence signals in a prototype-wise collaborative way.On the one hand, FedSC introduces an inter-contrastive learning strategy to bring instance-level embeddings closer to relational prototypes with the same semantics and away from distinct classes.On the other hand, FedSC devises consistent prototypes via a discrepancy aggregation manner, as a regularization penalty to constrain the optimization region of the local model.Moreover, a theoretical analysis for FedSC is provided to ensure a convergence guarantee.Experimental results on various challenging scenarios demonstrate the effectiveness of FedSC and the efficiency of crucial components.Our code is at https://github.com/hwang52/FedSC.

• We propose MVGFormer, a multi-view fusion framework with a dual-stream encoder guided by a visual graph. • We design two decoder variants: MF for hierarchical feature fusion and P3DA for multi-scale representation. • We introduce a view-masked self-supervised strategy that reconstructs masked views from remaining ones. • MVGFormer achieves superior performance over existing state-of-the-art cryo-ET segmentation methods. Cryo-Electron Tomography (cryo-ET) is a cutting-edge 3D imaging technology that enables detailed examination of biological macromolecular structures at near-atomic resolution. Recent deep learning applications on cryo-ET, such as cryo-ET segmentation, have drawn widespread interest for their potential to improve particle alignment, classification, and other tasks. However, current methods heavily rely on convolutional architectures, which prioritize local information while neglecting the global structural information inherent in cryo-ET data. Transformer-based models, known for their large receptive field, have become the de-facto design for 2D vision tasks due to their ability to effectively capture global information. This approach is also well-suited for 3D tasks, given the complex nature of 3D objects. Based on this, we extend 2D vision transformers into 3D and propose a novel transformer-based framework for cryo-ET segmentation, named MVGFormer. MVGFormer introduces a multi-view perspective fusion transformer encoder, which captures rich global structural information from multiple perspectives using unique positional embeddings. To enhance contextual awareness, we design a parallel context encoder that builds a visual graph to guide attention. We further introduce two complementary 3D decoders: multi-level feature fusion (MF) and parallel atrous convolutions (P3DA), which together capture multi-scale structural cues for precise segmentation. Furthermore, we introduce a view-masked self-supervised learning strategy to reinforce the effectiveness of the multi-view design and improve the model’s representation capability. To our knowledge, MVGFormer is the first transformer-based model for cryo-ET segmentation. We empirically evaluate MVGFormer on six cryo-ET datasets across three different tasks. Extensive experimental results demonstrate its superiority over state-of-the-art 3D segmentation methods.

Prediction of protein-ligand binding affinity (PLA) is a crucial problem in drug discovery. Recently, deep learning (DL) models have emerged as a promising and computationally efficient paradigm for the PLA prediction task, enabling rapid and scalable analysis while circumventing the time-consuming nature of experimental assays and the rigidity of conventional scoring functions. However, a significant domain knowledge gap often prohibits the effective integration of biological and computational insights, making it challenging to design deep learning models that comprehensively capture all relevant aspects. Training such models remains a complex undertaking involving multiple facets, including data heterogeneity, model interpretability, and biological plausibility. This review explores the key considerations for training DL models in PLA prediction task, including the choice of datasets, data processing techniques, model architecture design, model training strategies, and evaluation methodologies. Additionally, we discuss the potential applications of PLA prediction in traditional drug discovery and emerging areas, along with the challenges that currently hinder the optimal utilization of deep learning models in this field. This review aims to bridge the gap between computational biology and deep learning by providing a comprehensive guide for researchers interested in leveraging deep learning for PLA prediction.