Federated learning has been a popular distributed training paradigm that enables to train a shared model with data privacy protection. However, non-Independent Identically Distribution and long-tailed data distribution characteristics across mobile devices results in evident performance degradation, especially for classification tasks. Although plenty of research studies devote to alleviating classification performance degradation caused by highly-skewed data distribution, they still cannot improve the distinguishability of model representation on hard-to-learn tail classes, and face obvious divergence of local classifiers in FL setting. To this end, we propose Federated Classifier Representation Adjustment and Calibration to improve the representation distinguishability of tail classes and achieve inter-client representation alignment with acceptable resource consumption on attaching operations. We first design a Class Similarity-Aware Margin matrix to enlarge class representation discrepancy and improve local classifier discriminability on tail classes during client-side local training process. To mitigate the divergence of local classifiers across clients, we further propose the Self Distillation Classifier Calibration to achieve the aggregated global classifier calibration with the assistance of generated pseudo representation samples via self-distillation manner. We conduct various experiments under wide-range long-tailed and heterogeneous data settings. Experimental results show that FedCRAC outperforms state-of-the-art methods in terms of accuracy and resource consumption.

Federated Learning (FL) enables training Artificial Intelligence (AI) models over end devices without compromising their privacy. As computing tasks are increasingly performed by a combination of cloud, edge, and end devices, FL can benefit from this End-Edge-Cloud Collaboration (EECC) paradigm to achieve collaborative device-scale expansion with real-time access. Although Hierarchical Federated Learning (HFL) supports multitier model aggregation suitable for EECC, prior works assume the same model structure on all computing nodes, constraining the model scale by the weakest end devices. To address this issue, we propose Agglomerative Federated Learning (FedAgg), which is a novel EECC-empowered FL framework that allows the trained models from end, edge, to cloud to grow larger in size and stronger in generalization ability. FedAgg recursively organizes computing nodes among all tiers based on Bridge Sample Based Online Distillation Protocol (BSBODP), which enables every pair of parent-child computing nodes to mutually transfer and distill knowledge extracted from generated bridge samples. This design enhances the performance by exploiting the potential of larger models, with privacy constraints of FL and flexibility requirements of EECC both satisfied. Experiments under various settings demonstrate that FedAgg outperforms state-of-the-art methods by an average of 4.53% accuracy gains and remarkable improvements in convergence rate. Our code is available at https://github.com/wuzhiyuan2000/FedAgg.

Federated Learning (FL) is an emerging distributed learning paradigm with the privacy-preserving advantage of collaboratively training a shared model across multiple participants. Considering the prevailing device heterogeneity circumstance in practice, asynchronous interaction is introduced into FL to break the straggler barrier of synchronization, at the cost of significant accuracy degradation derived from model staleness. Although quite a few works attempt to partially mitigate the detrimental impact after occurring staleness issue, they neglect to control the overall staleness degree of clients-side local models from the whole training perspective, resulting in highly-stale models for aggregation and slow convergence speed. To this end, we propose a Staleness-Controlled Asynchronous Federated Learning (SC-AFL) method, which enables to restrict staleness degree of local models within a certain bound via dynamically tuning the aggregated strategy of each round, aiming to strike a good balance between accuracy guarantee and convergence acceleration. Specifically, we leverage the Lyapunov optimization framework to decouple the troublesome round-coupling problem into the single-round sequential solving problem, and further develop a deterministic algorithm that selects the aggregated number of clients to minimize training time under the constraint of maintaining staleness queue stability. Besides, we derive the theoretical convergence analysis of SC-AFL and also present the upper bound of the performance gap with the optimum. Extensive experiments on three datasets demonstrate the superiority of SC-AFL in terms of time-to-accuracy speedup on both IID and Non-IID data distributions, achieving a good balance between model accuracy and convergence efficiency in AFL system.