Federated learning is a promising paradigm to enable joint model training across distributed data while preserving data privacy. The distributed data are usually not identically and in-dependently distributed (Non-IID), which brings great challenges for federated learning. There have been existing work proposing to guide model aggregation between similar clients to deal with Non-IID data. But they typically assume a fully connected network topology, while new design issues need to be considered when it comes to a partially connected topology. In this work, we propose a probability-driven gossip framework for partially connected network topology with Non-IID data. The main idea is to discover similarity relationship between non-adjacent clients and guide the model exchange to encourage aggregation between similar clients. We explore cross-node similarity assessment and define probability to guide the model exchange and aggregation. Both similarity and communication cost are considered in the probability-driven gossip. Evaluation shows that the proposed scheme can achieve 13.04%-14.24% improvement in model accuracy, when compared with related work.

Federated learning (FL) is a distributed machine learning approach that allows multiple devices or computing nodes to jointly train models without sharing raw data. However, in real-world application scenarios, FL usually encounters a critical challenge of data heterogeneity. Recent studies have revealed that the client’s model suffers severe bias between the local model and global model, leading to global performance degradation. Improving the generalization of local learning would inherently reduce bias. It has been proved that self-paced learning on a single device can greatly achieve a better generalization result. However, it is not well explored how it can be applied to federated learning with a number of distributed nodes working cooperatively. Specifically, self-paced learning suggests using easy data and then gradually difficult data during model training. It is not straightforward to differentiate “easy” and “difficult” data at the local since global data distribution is not available, especially with severe data heterogeneity. To address the above issues, we propose a novel federated learning framework, Federation-Paced Learning (FedPL), which enables a self-paced process in federated learning and effectively improves the model performance. First, we propose schemes to analyze the data characteristics in terms of difficulty. Then we define a stage controller to synchronize the learning process across cooperative nodes to follow the easy-to-hard rule. Finally, we propose a client selection strategy to further improve the learning efficacy. We evaluate the performance of FedPL on several generic public datasets. Experiment results show that the proposed FedPL outperforms existing methods by up to 13.50% in terms of accuracy. Code is available at https://github.com/tnghua/FedPL.

Distributed machine learning methods are gaining significant attention for their ability to enhance computational efficiency and safeguard privacy. Federated learning and split learning are two prominent approaches in this domain. Recently, splitfed learning, a hybrid of both methods, was introduced to address their individual limitations. However, splitfed learning overlooks the non-IID (non-Independent and Identically Distributed) problem commonly encountered in distributed environments, which can lead to substantial degradation in model performance. In this paper, we introduce Clustered Splitfed Learning (CSFL), a novel approach that integrates clustering with splitfed learning. We propose two training processes tailored to the degree of data heterogeneity: Non-Personalized Clustered Splitfed Learning (NPCSFL) and Personalized Clustered Splitfed Learning (PCSFL). Our experimental results demonstrate that CSFL significantly improves both model accuracy and convergence rates.

Federated learning is proposed to train distributed data in a safe manner by avoiding to send data to server. The server maintains a global model and sends it to clients in each communication round, and then aggregates the updated local models to derive a new global model. Traditionally, the clients are randomly selected in each round and aggregation is based on weighted averaging. Researches show that the performance on IID data is satisfactory while significant accuracy drop can be observed for Non-IID data. In this paper, we explore the reasons and propose a novel aggregation approach for Non-IID data in federated learning. Specifically, we propose to group the clients according to classes of data they have, and select one set in each communication round. Local models from the same set are averaged as usual and the updated global model is sent to next group of clients for further training. In this way, the parameters are only averaged on similar clients and passed among different groups. Evaluation shows that the proposed scheme has advantages in terms of model accuracy and convergence speed with highly unbalanced data distribution and complex models.