In-context learning (ICL), a type of universal model, demonstrates exceptional generalization across a wide range of tasks without retraining by leveraging task-specific guidance from context, making it particularly effective for the intricate demands of neuroimaging. However, current ICL models, limited to 2D inputs and thus exhibiting suboptimal performance, struggle to extend to 3D inputs due to the high memory demands of ICL. In this regard, we introduce Neuroverse3D, an ICL model capable of performing multiple neuroimaging tasks in 3D (e.g., segmentation, denoising, inpainting). Neuroverse3D overcomes the large memory consumption associated with 3D inputs through adaptive parallel-sequential context processing and a U-shaped fusion strategy, allowing it to handle an unlimited number of context images. Additionally, we propose an optimized loss function to balance multi-task training and enhance focus on anatomical boundaries. Our study incorporates 43,674 3D multi-modal scans from 19 neuroimaging datasets and evaluates Neuroverse3D on 14 diverse tasks using held-out test sets. The results demonstrate that Neuroverse3D significantly outperforms existing ICL models and closely matches task-specific models, enabling flexible adaptation to medical center variations without retraining. The code and model weights are publicly available at https://github.com/jiesihu/Neuroverse3D.

In practical medical image segmentation tasks, ensuring confidence calibration is crucial. However, medical image segmentation typically relies on hard labels (one-hot vectors), and when minimizing the cross-entropy loss, the model’s softmax predictions are compelled to align with hard labels, resulting in over-confident predictions. To alleviate above problems, this study proposes a novel framework on calibration of medical image segmentation, called CALSeg. The Variational Label Smoothing (VLS) method is innovatively proposed, which learns the latent joint distribution of images and labels through variational inference to capture complex relationships between images and labels. This enables the effective estimation of latent soft labels by learning pixel-level information and semantic probability distribution features. The training of a neural network based on estimated soft labels provides a regularization effect, effectively preventing model overfitting and improving the calibration of the model. Comprehensive experiments on two medical image segmentation datasets demonstrate that CALSeg achieved optimal network calibration while also improving segmentation accuracy. The code is available at https://github.com/Guoxt/CALSeg.

Recent advancements in fMRI-based brain disorder diagnosis have shown that graph neural networks (GNNs) have been state-of-the-art methods for brain network analysis. Among them, transductive and inductive learning can be exploited by GNN. Transductive graphs, such as population graphs, take each subject as a node and use the node classification task for diagnosis. This line of work suffers from high computational costs and poor scalability to unseen data. Inductive methods, on the other hand, only consider labeled data and may suffer from overfitting and poor generalization when training with insufficient samples. To address these limitations, we propose a unified transductive-inductive network to study the properties of both transductive and inductive learning frameworks. Our approach is implemented in a self-knowledge distillation architecture, where transductive predictions are distilled from a transductive population graph network into an inductive network as a self-supervised regularization term. To preserve the topological properties within transductive graphs, i.e., inter-node similarity, we propose a topology-regularized self-knowledge distillation (Topo-KD) approach to regularize the student model’s learning. Evaluations on the ADNI dataset demonstrate the superiority of the approach in performance and scalability.

Source-free domain adaptation (SFDA) aims to adapt models trained on a labeled source domain to an unlabeled target domain without access to source data. In medical imaging scenarios, the practical significance of SFDA methods has been emphasized due to data heterogeneity and privacy concerns. Recent state-of-the-art SFDA methods primarily rely on self-training based on pseudo-labels (PLs). Unfortunately, the accuracy of PLs may deteriorate due to domain shift, thus limiting the effectiveness of the adaptation process. To address this issue, we propose a Chebyshev confidence guided SFDA framework to accurately assess the reliability of PLs and generate self-improving PLs for self-training. The Chebyshev confidence is estimated by calculating the probability lower bound of PL confidence, given the prediction and the corresponding uncertainty. Leveraging the Chebyshev confidence, we introduce two confidence-guided denoising methods: direct denoising and prototypical denoising. Additionally, we propose a novel teacher-student joint training scheme (TJTS) that incorporates a confidence weighting module to iteratively improve PLs' accuracy. The TJTS, in collaboration with the denoising methods, effectively prevents the propagation of noise and enhances the accuracy of PLs. Extensive experiments in diverse domain scenarios validate the effectiveness of our proposed framework and establish its superiority over state-of-the-art SFDA methods. Our paper contributes to the field of SFDA by providing a novel approach for precisely estimating the reliability of PLs and a framework for obtaining high-quality PLs, resulting in improved adaptation performance.

Recently, the study of multi-modal brain connectome has recorded a tremendous increase and facilitated the diagnosis of brain disorders. In this paradigm, functional and structural networks, e.g., functional and structural connectivity derived from fMRI and DTI, are in some manner interacted but are not necessarily linearly related. Accordingly, there remains a great challenge to leverage complementary information for brain connectome analysis. Recently, Graph Convolutional Networks (GNN) have been widely applied to the fusion of multi-modal brain connectome. However, most existing GNN methods fail to couple inter-modal relationships. In this regard, we propose a Cross-modal Graph Neural Network (Cross-GNN) that captures inter-modal dependencies through dynamic graph learning and mutual learning. Specifically, the inter-modal representations are attentively coupled into a compositional space for reasoning inter-modal dependencies. Additionally, we investigate mutual learning in explicit and implicit ways: (1) Cross-modal representations are obtained by cross-embedding explicitly based on the inter-modal correspondence matrix. (2) We propose a cross-modal distillation method to implicitly regularize latent representations with cross-modal semantic contexts. We carry out statistical analysis on the attentively learned correspondence matrices to evaluate inter-modal relationships for associating disease biomarkers. Our extensive experiments on three datasets demonstrate the superiority of our proposed method for disease diagnosis with promising prediction performance and multi-modal connectome biomarker location.

Medical image segmentation annotation suffers from annotator variation due to the inherent differences in annotators’ expertise and the inherent blurriness of medical images. In practice, using opinions from multiple annotators can effectively reduce the impact of such annotator-related biases. Meanwhile, it is common practice in deep learning to fuse multiple annotations through methods such as majority voting, but these methods ignore the rich information of annotator preferences ingrained in the original multi-annotator annotations. To address this issue, we propose a modeling annotator variation and annotator preference (AVAP) framework for multiple annotations medical image segmentation, which consists of three parts. First, the widely used encoder-decoder backbone network use to extract feature maps of the image. Second, an annotator variation modeling (AVM) module is devised to estimate the annotation variation among multiple annotators by modeling multi-annotations as a multi-class segmentation problem. Third, an annotator preference modeling (APM) module estimate each annotator’s preference-involved segmentation by annotator encoding and dynamic filter learning. The experiment on the RIGA benchmark with multiple annotations shows that our AVAP framework outperforms a range of state-of-the-art (SOTA) multiple annotations segmentation methods. Further, we are the first to introduce dynamic filter learning into the annotator preference modeling.

Multi-modal neuroimaging technology has greatly facilitated the diagnosis efficiency and diagnosis accuracy, and provides complementary information in discovering objective disease biomarkers. Conventional deep learning methods, e.g. convolutional neural networks, overlook relationships between nodes and fail to capture topological properties in graphs. Graph neural networks have been proven to be of great importance in modeling brain connectome networks and relating disease-specific patterns. However, most existing graph methods explicitly require known graph structures, which are not available in the sophisticated brain system. Especially in heterogeneous multi-modal brain networks, there exists a great challenge to model interactions among brain regions in consideration of inter-modal dependencies. In this study, we propose a Multimodal Dynamic Graph Convolution Network (MDGCN) for structural and functional brain network learning. Our method benefits from modeling inter-modal representations and relating attentive multi-model associations into dynamic graphs with a compositional correspondence matrix. Moreover, a bilateral graph convolution layer is proposed to aggregate multi-modal representations in terms of multi-modal associations. Extensive experiments on three datasets demonstrate the superiority of our proposed method in terms of disease classification, with the accuracy of 90.4%, 85.9% and 98.3% in predicting Mild Cognitive Impairment, Parkinson’s Disease, and Schizophrenia respectively. Our statistical evaluations on the correspondence matrix exhibit a high correspondence with previous evidence of biomarkers.

The brain age has been proven to be a phenotype of relevance to cognitive performance and brain disease. Achieving accurate brain age prediction is an essential prerequisite for optimizing the predicted brain-age difference as a biomarker. As a comprehensive biological characteristic, the brain age is hard to be exploited accurately with models using feature engineering and local processing such as local convolution and recurrent operations that process one local neighborhood at a time. Instead, Vision Transformers learn global attentive interaction of patch tokens, introducing less inductive bias and modeling long-range dependencies. In terms of this, we proposed a novel network for learning brain age interpreting with global and local dependencies, where the corresponding representations are captured by Successive Permuted Transformer (SPT) and convolution blocks. The SPT brings computation efficiency and locates the 3D spatial information indirectly via continuously encoding 2D slices from different views. Finally, we collect a large cohort of 22645 subjects with ages ranging from 14 to 97 and our network performed the best among a series of deep learning methods, yielding a mean absolute error (MAE) of 2.855 in validation set, and 2.911 in an independent test set.