Abstract Transformer models have demonstrated remarkable success in various fields, but face notable challenges in blood cell classification. These challenges arise primarily from two factors: (1) high computational demands due to the substantial number of parameters, and (2) insufficient labeled data in blood cell datasets, leading to overfitting risks. To address these issues, we introduce BccT, an innovative Transformer-based model specifically designed to optimize the classification of blood cell images. Central to our approach is the Token Fusion module, which intelligently merges similar tokens, thereby enhancing computational efficiency and reducing training overhead without compromising performance. This novel mechanism contrasts with traditional Transformer methods that process each token independently, resulting in significant redundancy. Additionally, we propose the Fixed-Random Classifier to mitigate overfitting risks, particularly in limited-data environments. Unlike conventional classifiers that necessitate constant parameter updates across all layers, our Fixed-Random Classifier comprises only three layers with predominantly static, randomly-initialized parameters, thus reducing dependency on extensive labeled data. Extensive experiments validate the effectiveness of our model. Using ViT_base as the backbone, our model achieves similar accuracy in half the training time compared to the ViT_base model. In experiments on four public blood cell datasets, our model improves accuracy by approximately 4.5% compared to other state-of-the-art methods.

Large Language Models (LLMs) have demonstrated exceptional performance across diverse tasks, yet their training remains highly resource intensive and susceptible to critical challenges such as training instability. A predominant source of this instability stems from gradient and loss spikes, which disrupt the learning process, often leading to costly interventions like checkpoint recovery and experiment restarts, further amplifying inefficiencies. This paper presents a comprehensive investigation into gradient spikes observed during LLM training, revealing their prevalence across multiple architectures and datasets. Our analysis shows that these spikes can be up to 1000× larger than typical gradients, substantially deteriorating model performance. To address this issue, we propose Spike-Aware Adam with Momentum Reset (SPAM), a novel optimizer designed to counteract gradient spikes through momentum reset and spike-aware gradient clipping. Extensive experiments, including both pre-training and fine-tuning, demonstrate that SPAM consistently surpasses Adam and its variants across a range of model scales. Additionally, SPAM facilitates memory-efficient training by enabling sparse momentum, where only a subset of momentum terms are maintained and updated. When operating under memory constraints, SPAM outperforms state-of-the-art memory-efficient optimizers such as GaLore and Adam-Mini. Our work underscores the importance of mitigating gradient spikes in LLM training and introduces an effective optimization strategy that enhances both training stability and resource efficiency at scale. Code is submitted.

Personalized federated learning (PFL) is a variant of federation learning, which improves the model performance of each participant through collaborative training while providing a customized local model that meets their unique needs. Balancing global (general) knowledge and local (personalized) knowledge is a core challenge in PFL. Existing PFL methods neglect knowledge forgetting during aggregation and updating, impacting the fusion of both types of knowledge. We empirically confirm the existence of knowledge forgetting, which leads to performance degradation during FL. This observation motivates us to rethink PFL from a knowledge perspective. Therefore, we propose Personalized Federated Learning Framework with Adaptive Model Fusion (pFedAMF). We consider the global model as global knowledge and history state models as local knowledge, and preserve them before forgetting occurs. After local training, we use an adaptive knowledge matrix to fuse knowledge from local, global, and history state models. This fused knowledge is then distilled into an individual model, thus transferring the fused knowledge into global models. Extensive experiments show that pFedAMF consistently outperforms FedAvg, achieving up to 5.22% average accuracy boost, and reducing computation cost by 82% and communication cost by 87%.

Building upon pre-trained ViT models, many advanced methods have achieved significant success in COVID-19 classification. Many scholars pursue better performance by increasing model complexity and parameters. While these methods can enhance performance, they also require extensive computational resources and extended training times. Additionally, the persistent challenge of overfitting, due to limited COVID-19 dataset sizes, remains a hurdle. To address these challenges, we proposed a novel method to optimize pre-trained transformer models for efficient COVID-19 classification with stochastic configuration networks (SCNs), referred to as OPT-CO. We proposed two optimization methods: sequential optimization (SeOp) and parallel optimization (PaOp), by incorporating optimizers in a sequential and parallel manner, respectively. Our method can enhance model performance without necessitating a significant parameter expansion. Additionally, we introduced OPT-CO-SCN to avoid overfitting problems through the adoption of random projection for head augmentation. The experiments were carried out to evaluate the performance of our proposed model based on two publicly available datasets. Based on the evaluation results, our method achieved superior, performance surpassing other state-of-the-art methods.

Abstract COVID-19 has caused over 6.35 million deaths and over 555 million confirmed cases till 11/July/2022. It has caused a serious impact on individual health, social and economic activities, and other aspects. Based on the gray-level co-occurrence matrix (GLCM), a four-direction varying-distance GLCM (FDVD-GLCM) is presented. Afterward, a five-property feature set (FPFS) extracts features from FDVD-GLCM. An extreme learning machine (ELM) is used as the classifier to recognize COVID-19. Our model is finally dubbed FECNet. A multiple-way data augmentation method is utilized to boost the training sets. Ten runs of tenfold cross-validation show that this FECNet model achieves a sensitivity of 92.23 ± 2.14, a specificity of 93.18 ± 0.87, a precision of 93.12 ± 0.83, and an accuracy of 92.70 ± 1.13 for the first dataset, and a sensitivity of 92.19 ± 1.89, a specificity of 92.88 ± 1.23, a precision of 92.83 ± 1.22, and an accuracy of 92.53 ± 1.37 for the second dataset. We develop a mobile app integrating the FECNet model, and this web app is run on a cloud computing-based client–server modeled construction. This proposed FECNet and the corresponding mobile app effectively recognize COVID-19, and its performance is better than five state-of-the-art COVID-19 recognition models.

COVID-19 pneumonia started in December 2019 and caused large casualties and huge economic losses. In this study, we intended to develop a computer-aided diagnosis system based on artificial intelligence to automatically identify the COVID-19 in chest computed tomography images. We utilized transfer learning to obtain the image-level representation (ILR) based on the backbone deep convolutional neural network. Then, a novel neighboring aware representation (NAR) was proposed to exploit the neighboring relationships between the ILR vectors. To obtain the neighboring information in the feature space of the ILRs, an ILR graph was generated based on the k-nearest neighbors algorithm, in which the ILRs were linked with their k-nearest neighboring ILRs. Afterward, the NARs were computed by the fusion of the ILRs and the graph. On the basis of this representation, a novel end-to-end COVID-19 classification architecture called neighboring aware graph neural network (NAGNN) was proposed. The private and public data sets were used for evaluation in the experiments. Results revealed that our NAGNN outperformed all the 10 state-of-the-art methods in terms of generalization ability. Therefore, the proposed NAGNN is effective in detecting COVID-19, which can be used in clinical diagnosis.