Federated Learning (FL) offers a decentralized approach to model training, where data remains local and only model parameters are shared between the clients and the central server. Traditional methods, such as Federated Averaging (FedAvg), linearly aggregate these parameters which are usually trained on heterogeneous data distributions, potentially overlooking the complex, high-dimensional nature of the parameter space. This can result in degraded performance of the aggregated model. While personalized FL approaches can mitigate the heterogeneous data issue to some extent, the limitation of linear aggregation remains unresolved. To alleviate this issue, we investigate the generative approach of diffusion model and propose a novel generative parameter aggregation framework for personalized FL, pFedGPA. In this framework, we deploy a diffusion model on the server to integrate the diverse parameter distributions and propose a parameter inversion method to efficiently generate a set of personalized parameters for each client. This inversion method transforms the uploaded parameters into a latent code, which is then aggregated through denoising sampling to produce the final personalized parameters. By encoding the dependence of a client's model parameters on the specific data distribution using the high-capacity diffusion model, pFedGPA can effectively decouple the complexity of the overall distribution of all clients' model parameters from the complexity of each individual client's parameter distribution. Our experimental results consistently demonstrate the superior performance of the proposed method across multiple datasets, surpassing baseline approaches.

The presence of noisy correspondence within cross-modal matching has significantly undermined the performance of existing matching methods. In this paper, we introduce a robust framework named Neighbor Auxiliary Corrector (NAC) for alleviating noise by utilizing the neighbors, which are indicative of similar textual targets. NAC is inspired by an observation that similar texts tend to correspond to similar images. Leveraging the zero-shot capabilities of Pre-trained Language Models (PLMs), we identify the top-k nearest neighbors for each positive image-text pair. Subsequently, the side information provided by these neighbors is harnessed for both sample verification and sample rectification. Extensive experiments on benchmark datasets demonstrate that our framework can significantly boost the performance and is more robust to various levels of noisy correspondence.

Deep hashing has been widely applied in large-scale data retrieval due to its superior retrieval efficiency and low storage cost. However, data are often scattered in data silos with privacy concerns, so performing centralized data storage and retrieval is not always possible. Leveraging the approach of federated learning (FL) to perform deep hashing is a recent research trend. However, existing frameworks mostly rely on the aggregation of the local deep hashing models, which are trained by performing similarity learning with local skewed data only. Therefore, they cannot work well for non-IID clients in a real federated environment. To overcome these challenges, we propose a novel federated hashing framework that enables participating clients to jointly train a shared deep hashing model by leveraging the class-wise prototypical hash codes. Globally, sharing global prototypes with only one prototypical hash code per class assists in learning consistent code distributions across clients while minimizing the cost of communication and privacy. Locally, the use of the global prototypical hash codes are maximized by jointly training a discriminator network and the local hashing network. Extensive experiments on benchmark datasets are conducted to demonstrate that our method can significantly improve the performance of the deep hashing model in the federated environments with non-IID data distributions.

The data heterogeneity has been a challenging issue in federated learning in both training and inference stages, which motivates a variety of approaches to learn either personalized models for participating clients or test-time adaptations for unseen clients. One such approach is employing a shared feature representation and a customized classifier head for each client. However, previous works either neglect the global head with rich knowledge or assume the new clients have enough labeled data, which significantly limit their broader practicality. In this work, we propose a lightweight framework to tackle with the label shift issue during the model deployment by test priors estimation and model prediction calibration. We also demonstrate the importance of training a balanced global model in FL so as to guarantee the general effectiveness of prior estimation approaches. Evaluation results on benchmark datasets demonstrate the superiority of our framework for model adaptation in unseen clients with unknown label shifts.

Gradient-based distributed learning techniques have been essential for machine model training on distributed samples without collecting raw data. However, such learning systems are vulnerable to both internal failures and external attacks. In this work, we study the Byzantine robustness of distributed learning over heterogeneous data for classification tasks, where each worker only contains data samples belonging to some specific classes (aka., label skew) and a fraction of workers are corrupted by a Byzantine adversary to conduct attacks by sending malicious gradients. Existing defenses usually fail under such heterogeneous cases. To remedy this, we propose a gradient decomposition scheme called DeSGD to achieve more robust distributed model training. The key idea for mitigating the impact of data heterogeneity on the Byzantine robustness is to divide the full global gradient into individual gradients of each data class and conduct resilient aggregation in a class-wise manner. The proposed framework can easily integrate existing advanced defense methods and local momentum mechanism. Evaluation results on the Fashion-MNIST dataset with various strong attacks demonstrate the improved robustness of learning over distributed data in the presence of both label skew and attacks.

Decentralized learning techniques have been increasingly popular for model training on distributed worker nodes. Unfortunately, such learning systems are vulnerable to failures and attacks. In this paper, we consider the decentralized learning problem over communication networks, in which worker nodes collaboratively train a machine learning model by exchanging model parameters with neighbors, but a fraction of nodes are corrupted by a Byzantine attacker and could conduct malicious attacks. Our key idea for mitigating Byzantine attacks is to check the direction and magnitude of the cross-update vectors (the difference between each received model and local model from the previous round) at each consensus round. We propose a similarity-based reweighting scheme to obtain a robust local model update for each worker. Our proposed method does not need to know the exact number of Byzantine nodes and can be employed in both static and time-varying networks. We evaluate our method on the Fashion-MNIST dataset with different Byzantine attacks and system sizes. Numerical results demonstrate the robustness of our proposed method against Byzantine attacks and superior performance than existing methods.

Distributed implementations of gradient-based algorithms have been essential for training large machine learning models on massive datasets. However, distributed learning algorithms are confronted with several challenges, including communication costs, straggler issues, and attacks from Byzantine adversaries. Existing works on attack-resilient distributed learning, e.g., the coordinate-wise median of gradients, usually neglect communication and/or straggler issues, and fail to defend against well-crafted attacks. Moreover, those methods are ineffective when more than half of workers are corrupted by a Byzantine adversary. To tackle those challenges simultaneously, we develop a robust gradient aggregation framework that is compatible with gradient compression and straggler mitigation techniques. Our proposed framework requires the parameter server to maintain an honest gradient as a reference at each iteration, thus can compute trust-score and similarity for each received gradient and tolerate arbitrary number of corrupted workers. We also provide convergence analysis of our method for non-convex optimization problems. Finally, experiments of image classification task on Fashion-MNIST dataset are conducted under various Byzantine attacks and gradient sparsification operations, and the numerical results demonstrate the effectiveness of our proposed strategy.

Gradient-based training in federated learning is known to be vulnerable to faulty/malicious clients, which are often modeled as Byzantine clients. To this end, previous work either makes use of auxiliary data at parameter server to verify the received gradients (e.g., by computing validation error rate) or leverages statistic-based methods (e.g. median and Krum) to identify and remove malicious gradients from Byzantine clients. In this paper, we remark that auxiliary data may not always be available in practice and focus on the statistic-based approach. However, recent work on model poisoning attacks has shown that well-crafted attacks can circumvent most of median- and distance-based statistical defense methods, making malicious gradients indistinguishable from honest ones. To tackle this challenge, we show that the element-wise sign of gradient vector can provide valuable insight in detecting model poisoning attacks. Based on our theoretical analysis of the Little is Enough attack, we propose a novel approach called SignGuard to enable Byzantine-robust federated learning through collaborative malicious gradient filtering. More precisely, the received gradients are first processed to generate relevant magnitude, sign, and similarity statistics, which are then collaboratively utilized by multiple filters to eliminate malicious gradients before final aggregation. Finally, extensive experiments of image and text classification tasks are conducted under recently proposed attacks and defense strategies. The numerical results demonstrate the effectiveness and superiority of our proposed approach.

Federated learning is an emerging technique to collaboratively train machine learning models over multiple clients without exposing private data but suffers from heterogeneous data distributions across clients, which results in slow convergence and degraded model performance. This challenge has motivated several approaches to learn a personalized model that has better model accuracy than the global model for each participating client. On the other hand, the success of multi-task learning suggests that data in similar tasks often share a common feature representation, while the output layer (classifier) for classification is more task-correlated. Existing methods usually neglect the collaboration between locally trained classifiers, which makes each client failed to fully benefit from data in other clients. Therefore, we perform a linear combination based classifier collaboration method for achieving better personalized model performance. Moreover, training a local feature extractor based on local data is prone to over-fitting when local data is insufficient and cannot be fully corrected by global aggregation. To tackle this issue, we propose a feature-regularized training strategy to mitigate the local over-fitting risk and reduce the parameter divergence across clients, facilitating the global feature extractor aggregation. With extensive experiments performed on Fashion-MNIST and CIFAR-10 datasets, we demonstrate the performance improvement and robustness of our method over existing methods.

In this paper, we study the information transmission problem under the distributed learning framework, where each worker node is merely permitted to transmit a m-dimensional statistic to improve learning results of the target node. Specifically, we evaluate the corresponding expected population risk (EPR) under the regime of large sample sizes. We prove that the performance can be enhanced since the transmitted statistics contribute to estimating the underlying distribution under the mean square error measured by the EPR norm matrix. Accordingly, the transmitted statistics correspond to the eigenvectors of this matrix, and the desired transmission allocates these eigenvectors among the statistics such that the EPR is minimal. Moreover, we provide the analytical solution of the desired statistics for single-node and two-node transmission, where a geometrical interpretation is given to explain the eigenvector selection. For the general case, an efficient algorithm that can output the allocation solution is developed based on the node partitions.

The training efficiency of distributed learning systems is vulnerable to stragglers, namely, those slow worker nodes. A naive strategy is performing the distributed learning by incor-porating the fastest K workers and ignoring these stragglers, which may induce high deviation for non-IID data. To tackle this, we develop a Live Gradient Compensation (LGC) strategy to incorporate the one-step delayed gradients from stragglers, aiming to accelerate learning process and utilize the stragglers simultaneously. In LGC framework, mini-batch data are divided into smaller blocks and processed separately, which makes the gradient computed based on partial work accessible. In addition, we provide theoretical convergence analysis of our algorithm for non-convex optimization problem under non-IID training data to show that LGC-SGD has almost the same convergence error as full synchronous SGD. The theoretical results also allow us to quantify a novel tradeoff in minimizing training time and error by selecting the optimal straggler threshold. Finally, extensive simulation experiments of image classification on CIFAR-10 dataset are conducted, and the numerical results demonstrate the effectiveness of our proposed strategy.