This study addresses the core challenges in federated learning (FL), namely achieving optimal model utility, safeguarding local data privacy, and maintaining efficient communication. While previous research has focused on either the privacy-utility or communication-utility trade-offs, the investigation of simultaneously considering utility, privacy protection, and communication efficiency has been largely overlooked. In this paper, we propose a novel training framework for FL that combines communication efficiency and differential privacy. Specifically, we employ quantization and binomial noise on model updates to enhance privacy protection and communication efficiency concurrently. Through convergence and privacy analysis, we formulate an optimization problem that maximizes model utility while adhering to privacy and communication constraints. Additionally, we introduce an adaptive algorithm to determine key system parameters, including the level of quantization and privacy noise. Simulation results validate the effectiveness of our proposed FL framework and parameter optimization algorithm.

Federated learning (FL) is a promising paradigm for privacy-preserving deep learning using data distributed on Internet of Things devices. Traditional model sharing-based methods, e.g., federated averaging (FedAvg), suffer from high communication overhead and difficulty in accommodating heterogeneous model architectures. Federated distillation (FD) is a recently proposed alternative to enable communication-efficient and robust FL, as well as heterogeneous client models. However, there is a lack of theoretical understanding of FD-based methods, and their design guidelines remain elusive. This article presents a generic meta-algorithm for FD, generalizing most existing FD training algorithms. By studying a linear classification problem, we show that, with sufficient distillation samples, the training performance of the meta-algorithm is the same as the vanilla FedAvg. To guide the algorithm design and improve communication efficiency, we further investigate the binary classification problem with a Gaussian mixture model, which shows that more distillation data and sampling data with higher confidence improve the training performance. Furthermore, we propose an effective distillation data sampling technique to improve the performance of the FD-meta algorithm, which also reduces communication overhead. Simulations on the benchmark data sets validate the theoretical findings and demonstrate that our proposed algorithm effectively reduces the communication overhead while achieving a satisfactory performance.

Federated learning (FL) is a privacy-preserving collaborative learning framework, and differential privacy can be applied to further enhance its privacy protection. Existing FL systems typically adopt Federated Average (FedAvg) as the training algorithm and implement differential privacy with a Gaussian mechanism. However, the inherent privacy-utility trade-off in these systems severely degrades the training performance if a tight privacy budget is enforced. Besides, the Gaussian mechanism requires model weights to be of high-precision. To improve communication efficiency and achieve a better privacy-utility trade-off, we propose a communication-efficient FL training algorithm with differential privacy guarantee. Specifically, we propose to adopt binary neural networks (BNNs) and introduce discrete noise in the FL setting. Binary model parameters are uploaded for higher communication efficiency and discrete noise is added to achieve the client-level differential privacy protection. The achieved performance guarantee is rigorously proved, and it is shown to depend on the level of discrete noise. Experimental results based on MNIST and Fashion-MNIST datasets will demonstrate that the proposed training algorithm achieves client-level privacy protection with performance gain while enjoying the benefits of low communication overhead from binary model updates.

Federated learning (FL) is a powerful distributed machine learning framework where a server aggregates models trained by different clients without accessing their private data. Hierarchical FL, with a client-edge-cloud aggregation hierarchy, can effectively leverage both the cloud server’s access to many clients’ data and the edge servers’ closeness to the clients to achieve a high communication efficiency. Neural network quantization can further reduce the communication overhead during model uploading. To fully exploit the advantages of hierarchical FL, an accurate convergence analysis with respect to the key system parameters is needed. Unfortunately, existing analysis is loose and does not consider model quantization. In this paper, we derive a tighter convergence bound for hierarchical FL with quantization. The convergence result leads to practical guidelines for important design problems such as the client-edge aggregation and edge-client association strategies. Based on the obtained analytical results, we optimize the two aggregation intervals and show that the client-edge aggregation interval should slowly decay while the edge-cloud aggregation interval needs to adapt to the ratio of the client-edge and edge-cloud propagation delay. Simulation results shall verify the design guidelines and demonstrate the effectiveness of the proposed aggregation strategy.

Federated learning (FL) is a promising paradigm to enable privacy-preserving deep learning from distributed data. Most previous works are based on federated average (FedAvg), which, however, faces several critical issues, including a high communication overhead and the difficulty in dealing with heterogeneous model architectures. Federated Distillation (FD) is a recently proposed alternative to enable communication-efficient and robust FL, which achieves orders of magnitude reduction of the communication overhead compared with FedAvg and is flexible to handle heterogeneous models at the clients. However, so far there is no unified algorithmic framework or theoretical analysis for FD-based methods. In this paper, we first present a generic meta-algorithm for FD and investigate the influence of key parameters through empirical experiments. Then, we verify the empirical observations theoretically. Based on the empirical results and theory, we propose a communication-efficient FD algorithm with active data sampling to improve the model performance and reduce the communication overhead. Empirical simulations on benchmark datasets will demonstrate that our proposed algorithm effectively and significantly reduces the communication overhead while achieving a satisfactory performance.

Federated Learning is a collaborative machine learning framework to train a deep learning model without accessing clients’ private data. Previous works assume one central parameter server either at the cloud or at the edge. The cloud server can access more data but with excessive communication overhead and long latency, while the edge server enjoys more efficient communications with the clients. To combine their advantages, we propose a client-edge-cloud hierarchical Federated Learning system, supported with a HierFAVG algorithm that allows multiple edge servers to perform partial model aggregation. In this way, the model can be trained faster and better communication-computation trade-offs can be achieved. Convergence analysis is provided for HierFAVG and the effects of key parameters are also investigated, which lead to qualitative design guidelines. Empirical experiments verify the analysis and demonstrate the benefits of this hierarchical architecture in different data distribution scenarios. Particularly, it is shown that by introducing the intermediate edge servers, the model training time and the energy consumption of the end devices can be simultaneously reduced compared to cloud-based Federated Learning.