Federated Learning (FL) has emerged as a pivotal paradigm for multi-client collaborative learning, primarily due to its inherent capability to safeguard privacy. Nonetheless, the heterogeneity among FL clients, characterized by their disparate resource capabilities and not independent and identical (NonIID) local datasets, presents a significant challenge. Specifically, low-resource clients, such as edge devices, grapple with the inadequacy to accommodate the entire model parameter set for training purposes. To mitigate this issue, preceding research has ventured into devising methodologies that entail extracting sub-models from the overarching global model, tailored to the specific communication, computational, and memory constraints of individual clients. Despite these advancements, prevailing sub-model extraction techniques, which predominantly hinge on pre-established rules, overlook a crucial factor: the impact of NonIID local data on the trajectory of neuron update dynamics. This oversight can amplify discrepancies between the practical local updates inferred by the sub-model and those anticipated via the entire model, thereby undermining overall performance. In this paper, we introduceFedGSE, an innovative Gradient-based Neuron Selection methodology designed explicitly for FL environments. This methodology aims to curate sub-models that significantly reduce discrepancies in local updates, enhancing alignment with the global model's learning trajectory. Central to theFedGSEapproach is a sophisticated algorithm that handpicks critical neurons for sub-model construction. These neurons are identified through their pronounced gradient magnitudes, resulting from the training of the global model on a dataset mirroring the client's data distribution. Consequently, the sub-model's induced local gradient updates closely emulate those derived from directly training the client's data on the full global model, fostering enhanced alignment and performance. Extensive experiments over diverse datasets and tasks demonstrate the superiority ofFedGSEover existing baselines.

Recent advancements in Large Language Models (LLMs) and Large Multi-modal Models (LMMs) have shown potential in various medical applications, such as Intelligent Medical Diagnosis. Although impressive results have been achieved, we find that existing benchmarks do not reflect the complexity of real medical reports and specialized in-depth reasoning capabilities. In this work, we establish a comprehensive benchmark in the field of medical specialization and introduced RJUA-MedDQA, which contains 2000 real-world Chinese medical report images poses several challenges: comprehensively interpreting imgage content across a wide variety of challenging layouts, possessing the numerical reasoning ability to identify abnormal indicators and demonstrating robust clinical reasoning ability to provide the statement of disease diagnosis, status and advice based on a collection of medical contexts. We carefully design the data generation pipeline and proposed the Efficient Structural Restoration Annotation (ESRA) Method, aimed at restoring textual and tabular content in medical report images. This method substantially enhances annotation efficiency, doubling the productivity of each annotator, and yields a 26.8% improvement in accuracy. We conduct extensive evaluations, including few-shot assessments of 5 LMMs which are capable of solving Chinese medical QA tasks. To further investigate the limitations and potential of current LMMs, we conduct comparative experiments on a set of strong LLMs by using image-text generated by ESRA method. We report the performance of baselines and offer several observations: (1) The overall performance of existing LMMs is still limited; however LMMs more robust to low-quality and diverse-structured images compared to LLMs. (3) Reasoning across context and image content present significant challenges. We hope this benchmark helps the community make progress on these challenging tasks in multi-modal medical document understanding and facilitate its application in healthcare. Our dataset will be publicly available for noncommercial use at https://github.com/Alipay-Med/medDQA_benchmark.git

Sub-model extraction based federated learning has emerged as a popular strategy for training models on resource-constrained devices. However, existing methods treat all clients equally and extract sub-models using predetermined rules, which disregard the statistical heterogeneity across clients and may lead to fierce competition among them. Specifically, this paper identifies that when making predictions, different clients tend to activate different neurons of the entire model related to their respective distributions. If highly activated neurons from some clients with one distribution are incorporated into the sub-model allocated to other clients with different distributions, they will be forced to fit the new distributions, which can hinder their activation over the previous clients and result in a performance reduction. Motivated by this finding, we propose a novel method called FedDSE, which can reduce the conflicts among clients by extracting sub-models based on the data distribution of each client. The core idea of FedDSE is to empower each client to adaptively extract neurons from the entire model based on their activation over the local dataset. We theoretically show that FedDSE can achieve an improved classification score and convergence over general neural networks with the ReLU activation function. Experimental results on various datasets and models show that FedDSE outperforms all state-of-the-art baselines.