Federated learning (FL) trains machine learning models over heterogeneous and decentralized datasets. Communication between client and server can be a major bottleneck for FL, especially in cases of large models. Moreover, real-world FL problems often involve data heterogeneity issues such as class imbalance. We propose an approach to address these issues of class balance and communication efficiency in Federated Learning. Our strategy is based on two key elements: 1. a novel adaptive voting-based federated dropout on client models addressing communication bottlenecks and data heterogeneity, and 2. a heterogeneous quantization method that can adjust to clients' bandwidth requirements. We conduct experiments across several datasets and models demonstrating that these two components work together to balance the trade-off between communication costs and model performance with clients having heterogeneous communication bandwidth. Importantly, our approach improves performance on imbalanced datasets like CIFAR-10-LT and CIFAR-100-LT, which is critical for addressing class imbalance in federated learning. On CIFAR-10 We get approximately a seven-factor reduction in communication cost without degrading the quality of the model.

Neural networks are often pre-trained on a large source dataset and then fine-tuned on a smaller target dataset. Although pre-training on large-scale datasets is very useful, it has a few disadvantages, such as (1) high training cost and (2) domain mismatch where pre-training on a less-related source might lead to poor results in a target model. Examples of this are areas like underwater imaging, medical imaging, microscopic imaging, etc. Many datasets in these domains also have class imbalance which makes transfer learning less effective. In this paper, we propose an efficient method for selective pre-training, i.e. selecting relevant subsets from a pre-training dataset. Fine-tuning with our method gives better accuracy while increasing training efficiency. We validate our technique with selective pre-training on ImageNet21k and ImageNet1k datasets, and fine-tuning on tasks like image classification and image segmentation. We conduct experiments on several imbalanced datasets and compare our performance with full pre-training as well as other state-of-the-art methods to handle class imbalance. On imbalanced CIFAR-10 we get an accuracy of 77% with pre-training on 500k images of ImageNet1k compared to 74% for full pre-training on ImageNet.