Data heterogeneity impacts the performance of Federated Learning (FL) by introducing training noise. Although representative client sampling can help mitigate the issue, it remains challenging to implement without compromising data privacy. This work introduces a new method to address the problem by proposing an affordable blind (privacy preserving) clustering mechanism for conducting stratified client sampling. Inspired by the ‘dialect quiz’, we propose a ‘response test’ to cluster clients whose models have learned similar high-level features. This approach facilitates representative client sampling without the need for direct access to client data. We demonstrate empirically that our method yields client samples with low relative entropy with respect to the global data distribution, indicating increased representativeness. Convergence experiments reveal that applying our method significantly improves the convergence and accuracy of the global model compared to strong baselines like SCAFFOLD and FL-CIR. Additionally, the reduced number of training rounds required to achieve target accuracy leads to decreased communication overhead and computational expense, making our approach promising for practical FL implementations.

Recently, local peer topology has been shown to influence the overall convergence of decentralized learning (DL) graphs in the presence of data heterogeneity. In this paper, we demonstrate the advantages of constructing a proxy-based locally heterogeneous DL topology to enhance convergence and maintain data privacy. In particular, we propose a novel peer clumping strategy to efficiently cluster peers before arranging them in a final training graph. By showing how locally heterogeneous graphs outperform locally homogeneous graphs of similar size and from the same global data distribution, we present a strong case for topological pre-processing. Moreover, we demonstrate the scalability of our approach by showing how the proposed topological pre-processing overhead remains small in large graphs while the performance gains get even more pronounced. Furthermore, we show the robustness of our approach in the presence of network partitions.

Federated learning (FL) facilitates the training and deploying AI models on edge devices. Preserving user data privacy in FL introduces several challenges, including expensive communication costs, limited resources, and data heterogeneity. In this paper, we propose SPATL, an FL method that addresses these issues by: (a) introducing a salient parameter selection agent and communicating selected parameters only; (b) splitting a model into a shared encoder and a local predictor, and transferring its knowledge to heterogeneous clients via the locally customized predictor. Additionally, we leverage a gradient control mechanism to further speed up model convergence and increase robustness of training processes. Experiments demonstrate that SPATL reduces communication overhead, accelerates model inference, and enables stable training processes with better results compared to state-of-the-art methods. Our approach reduces communication cost by up to 86.45%, accelerates local inference by reducing up to 39.7% FLOPs on VGG-11, and requires 7.4× less communication overhead when training ResNet-20.11Code is available at: https://github.com/yusx-swapp/SPATL