Multimodal fusion is an effective solution for holistic brain tumor diagnosis; however, it faces challenges under missing modalities. Traditional multi-encoder architectures can easily capture modality-specific features, while single-encoder architectures readily obtain modality-shared features. The reverse, however, is challenging. In this paper, we propose a two-stage dual-view prototype learning framework to extract the modality-specific feature and the class-specific feature simultaneously. In the first stage, we utilize the Transformer decoder to learn the modality-prototypes that are used to optimize the modality reconstruction task. A masked autoencoder is introduced to generate shared features of incomplete modalities. The learned modality-prototypes that contain modality-specific features are blended with the modality-share features for the reconstruction process. In the second stage, we learn the class-prototypes through the Transformer decoder to generate a segmentation mask through voxel-to-prototype comparison. A masked modality strategy is introduced to handle random modality absence during training. Furthermore, modality-view and class-view contrastive learning strategies are developed to enhance prototype learning. We conduct experiments on BraTS2020 and BraTS2018; the experimental results demonstrate the superior performance of our model under various missing modality scenarios. On BraTS2020, our model achieves DSC improvements of 5.9% for ET, 0.5% for TC, and 0.2% for WT compared to state-of-the-art methods. Notably, in the challenging T1C modality missing scenario, our model achieved clinically significant gains of 9.5% for ET and 1.8% for TC. The code is available at https://github.com/Xiheran/MCPL.

In the Industrial Internet of Things (IIoT), collaborative intelligence among distributed devices is essential for achieving autonomy and robustness, especially when facing non-IID and out-of-distribution (OOD) data. Deep neural networks have achieved significant success in various applications, but their prediction confidence often degrades on OOD data, which is critical in IIoT environments with heterogeneous sources. Centralized OOD detection methods assume data is centrally stored and require a large number of real OOD samples, which are impractical and costly in federated learning due to data silos and privacy issues. To address the above challenges, we formulate the new problem of OOD detection on heterogeneous data in a federated learning framework. We propose a novel multi-task optimal model named FOOD that improves OOD accuracy through optimal transport theory in a distributed manner with data privacy protection. Specifically, FOOD generates pseudo-OOD samples based on optimal transport theory and purifies training samples to enhance classification accuracy. We use the Wasserstein distance to measure the similarity between in-distribution and out-of-distribution samples and generate heterogeneous pseudo-OOD samples among different clients. FOOD is a plug-and-play plugin that can improve deep neural models' performance without introducing extra overhead. Experiments on OOD datasets show that FOOD significantly enhances OOD detection and classification on several public OOD datasets, AUROC improved by 1.69%, AUPR by 1.75%, and ACC by 4.67%. Using optimal transport theory, our work provides a practical approach to improving data generalization in generative models.

Federated domain generalization (FDG) enables collaborative learning across distributed devices to build a global prediction model capable of generalizing to diverse environments. While existing methods perform well on homogeneous data distributions, they struggle with performance degradation caused by data drift in heterogeneous settings. To address this challenge, we propose FedFAT, a novel method that leverages frequency domain adaptive interpolation to mitigate data drift effectively. FedFAT allows each client to adaptively exchange amplitude information for improved generalization while retaining phase information locally to ensure privacy. The interpolation process-including amplitude interpolation size, mask position, and fusion ratio-is dynamically determined based on the difference between the client's amplitude and that of the shared library. Additionally, we introduce amplitude normalization to align features across clients by batch-normalizing images from multi-source distributions, thereby reducing the negative impact of data drift. Building on these uniform features, weight perturbation is applied to ensure consistent low loss for local models. Extensive experiments and ablation studies on medical image analysis tasks, including MRI prostate segmentation and breast cancer tissue classification, demonstrate the superiority of FedFAT in handling data drift and improving generalization performance. Our results underscore the critical role of frequency domain adaptive interpolation in addressing data drift and enhancing the robustness of federated domain generalization.

Semi-supervised learning (SSL) has been prevailed in medical image analysis field because it leverage unlabeled data for training powerful models without incurring extra high annotation cost. However, the existing SSL models face the challenge of performance degradation in open-set scenario where both in-distribution (ID) and out-of-distribution (OOD) samples are mixed in unlabeled data. The existing two-phases methods treat OOD detection and semi-supervised classification as two independent tasks which fail to reveal their mutual reinforcement ability. In this research, we introduce a joint optimization framework designed to enhance both the semi-supervised classification task and the OOD detection task through an iterative process. Specifically, our approach employs a unified model to assess the likelihood of images being OOD sample, subsequently filtering these instances from the pool of unlabeled data. The model parameters update and the OOD detection are optimized alternately. Additionally, to avoid the model overconfidence, we introduce logit normalization (Logit-Norm) loss to provide more reliable predictions. To validate the effectiveness of our method, we use the ISIC2018 dataset as the ID dataset and mix OOD samples from other medical image datasets to train classification model. Experimental results demonstrate that our method successfully mitigates the influence of OOD data on semi-supervised medical image classification performance while also improving OOD detection performance. The proposed framework successfully addresses the challenges posed by OOD samples in semi-supervised learning, offering a promising solution for medical image classification tasks that involve OOD data.

Transformers have shown great potential in vision tasks such as semantic segmentation. However, most of the existing transformer-based segmentation models neglect the cross-attention between pixel features and class features which impedes the application of transformers. Inspired by the concept of object queries in k-means Mask Transformer, we develop cluster learning and contrastive cluster assignment (CCA) for medical image segmentation in this paper. The cluster learning leverages the object queries to fit the feature-level cluster centers. The contrastive cluster assignment is introduced to guide the pixel class prediction using the cluster centers. Our method is a plug-in and can be integrated into any model. We design two networks for supervised segmentation tasks and semi-supervised segmentation tasks respectively. We equip the decoder with our proposed modules for the supervised segmentation to improve the pixel-level predictions. For the semi-supervised segmentation, we enhance the feature extraction capability of the encoder by using our proposed modules. We conduct comprehensive comparison and ablation experiments on public medical image datasets (ACDC, LA, Synapse, and ISIC2018), the results demonstrate that our proposed models outperform state-of-the-art models consistently, validating the effectiveness of our proposed method. The source code is accessible at https://github.com/zhujinghua1234/CCA-Seg.

The emergence of lightweight models provides a new solution for the classification of Diabetic Retinopathy (DR). These lightweight models have a fewer parameters and low computational complexity. However, existing lightweight models in Diabetic Retinopathy (DR) still face challenges: Convolutional Neural Networks (CNNs) are limited by their receptive fields and lack modeling capabilities over long distances. Although the models based on Transformer can capture global information, the usually have high computational complexity. Recent lightweight models that try to combine the strengths of CNN and Transformer tend to have a simple serial architecture that fails to take full advantage of both. To address these challenges, we propose a new lightweight network, MobileP2VT, for the classification of diabetic retinopathy. The model uses lightweight convolution and Transformer parallel processing to capture local details and global information of the image. In addition, we have designed a new attention mechanism, Sparse Token Attention (SPTA), which reduces the amount of attention computation in Transformer. Experimental results on Eyepacs and APTOS 2019 datasets demonstrate the high efficiency of MobileP2VT.

With the increasing scale of data to be processed by the recommendation system and the rapid growth of the number of users and commodities, the problems of data sparsity and cold start become more and more severe. Many studies have found that introducing implicit social relationships into recommendation can alleviate the above problems. Among these studies, the method of identifying implicit friends for each user in heterogeneous information networks based on meta-paths achieves good performance. However, meta paths need to be preset artificially and require strong professional domain knowledge, which makes it impossible to capture semantic information accurately and effectively. In this paper, we propose a novel social recommendation model termed as SILK-BPR which is an extension of the Self Guided Walk(SILK) model to by bypass the meta-paths and identify implicit friends for recommendation. Specially, SILK-BPR first establishes a guidance matrix to record the transition probability from user nodes to interest nodes. And then conducts a self-guided walking on the matrix to mine the implicit friends which are classified into different types according to the appear times in different sequences. Finally, these implicit friends are integrated into an enhanced social Bayesian Personalized Ranking model for Top-N recommendation. Experimental results on real-world datasets demonstrate that SILK-BPR significantly outperforms state-of-the-art methods.