Federated learning (FL) allows multiple data-owners to collaboratively train machine learning models by exchanging local gradients, while keeping their private data on-device. To simultaneously enhance privacy and training efficiency, recently parameter-efficient fine-tuning (PEFT) of large-scale pretrained models has gained substantial attention in FL. While keeping a pretrained (backbone) model frozen, each user fine-tunes only a few lightweight modules to be used in conjunction, to fit specific downstream applications. Accordingly, only the gradients with respect to these lightweight modules are shared with the server. In this work, we investigate how the privacy of the fine-tuning data of the users can be compromised via a malicious design of the pretrained model and trainable adapter modules. We demonstrate gradient inversion attacks on a popular PEFT mechanism, the adapter, which allow an attacker to reconstruct local data samples of a target user, using only the accessible adapter gradients. Via extensive experiments, we demonstrate that a large batch of fine-tuning images can be retrieved with high fidelity. Our attack highlights the need for privacy-preserving mechanisms for PEFT, while opening up several future directions. Our code is available at https://github.com/info-ucr/PEFTLeak.

Secure aggregation is an information-theoretic mechanism for gradient aggregation in federated learning, to aggregate the local user gradients without revealing them in the clear. In this work, we study secure aggregation under gradient sparsification constraints, for resource-limited wireless networks, where only a small fraction of local parameters are aggregated from each user during training (as opposed to the full gradient). We demonstrate that conventional mechanisms can reveal sensitive user data when aggregating sparsified gradients, due to the auxiliary coordinate information shared during sparsification, even when the individual gradients are not disclosed in the clear. We then propose a coordinate-hiding sparsified secure aggregation mechanism to address this challenge, which hides both the gradient parameters and the associated coordinates under formal information-theoretic privacy guarantees. Our framework reduces the communication overhead of conventional secure aggregation baselines by an order of magnitude without compromising model accuracy.

Secure aggregation (SA) is a privacy-enhancing framework for federated learning, to aggregate the local gradient updates from the users without revealing them in the clear. Conventional SA frameworks are built under the assumption of homogeneous computational resources across the users, where users are bound to train a local model whose dimensions are as large as the global model, preventing resource-limited users from participating in training. In this work, we propose a novel secure submodel training framework to address this challenge, where users train and communicate partial submodels through an adaptable secure aggregation mechanism during training. Our framework enables the participation of all users with varying computation and communication resources, while ensuring formal information-theoretic privacy guarantees for the individual local updates.

Privacy-preserving coded computing is a popular framework for multiple data-owners to jointly train machine learning models, with strong end-to-end information-theoretic privacy guarantees for the local data. A major challenge against the scalability of current approaches is their communication overhead, which is quadratic in the number of users. Towards addressing this challenge, we present SCALR, a communication-efficient collaborative learning framework for training logistic regression models. To do so, we introduce a novel coded computing mechanism, by decoupling the communication-intensive encoding operations from real-time training, and offloading the former to a data-independent offline phase, where the communicated variables are independent from training data. As such, the offline phase can be executed proactively during periods of low network activity. Communication complexity of the data-dependent (online) training operations is only linear in the number of users, greatly reducing the quadratic state-of-the-art. Our theoretical analysis presents the information-theoretic privacy guarantees, and shows that SCALR achieves the same performance guarantees as the state-of-the-art, in terms of adversary resilience, robustness to user dropouts, and model convergence. Through extensive experiments, we demonstrate up to$80\times $reduction in online communication overhead, and$6\times $speed-up in the wall-clock training time compared to the state-of-the-art.

Clustered federated learning is a popular paradigm to tackle data heterogeneity in federated learning, by training personalized models for groups of users with similar data distributions. A critical challenge is to protect the privacy of individual user updates, as the latter can reveal extensive information about sensitive local datasets. To do so, a recent promising approach is information-theoretic secure aggregation, where parties learn the aggregate (sum) of user updates, but no further information is revealed about the individual updates. In this work, we present the first single-server secure aggregation framework in the context of clustered federated learning, to learn the aggregate of user updates for any clustering of users, but without learning any information about the local updates or cluster identities. Our framework can achieve linear communication complexity under formal information-theoretic privacy guarantees, while providing key trade-offs between communication and computation complexity, adversary tolerance, and resilience to user dropouts.

Secure aggregation is an information-theoretic mechanism for gradient aggregation in federated learning, to aggregate the local user gradients without revealing them in the clear. In this work, we study secure aggregation under gradient sparsification constraints, for resource-limited wireless networks, where only a small fraction of local parameters are aggregated from each user during training (as opposed to the full gradient). We first identify the vulnerabilities of conventional secure aggregation mechanisms under gradient sparsification. We show that conventional mechanisms can reveal sensitive user data when aggregating sparsified gradients, due to the auxiliary coordinate information shared during sparsification, even when the individual gradients are not disclosed in the clear. We then propose TinySecAgg, a novel coordinate-hiding sparsified secure aggregation mechanism to address this challenge, under formal information-theoretic privacy guarantees. Our framework reduces the communication overhead of conventional secure aggregation baselines by an order of magnitude (up to 22.5×) without compromising model accuracy.

Collaborative machine learning enables privacy-preserving training of machine learning models without collecting sensitive client data. Despite recent breakthroughs, communication bottleneck is still a major challenge against its scalability to larger networks. To address this challenge, in this work we propose PICO, the first collaborative learning framework with linear communication complexity, significantly improving over the quadratic state-of-the-art, under formal information-theoretic privacy guarantees. Theoretical analysis demonstrates that PICO slashes the communication cost while achieving equal computational complexity, adversary resilience, robustness to client dropouts, and model accuracy to the state-of-the-art. Extensive experiments demonstrate up to 91× reduction in the communication overhead, and up to 8× speed-up in the wall-clock training time compared to the state-of-the-art. As such, PICO addresses a key technical challenge in multi-party collaborative learning, paving the way for future large-scale privacy-preserving learning frameworks.

Over-the-air federated learning (FL) is a recent paradigm to address the communication bottleneck of FL, where a machine learning model is trained by aggregating the local gradients directly in the wireless medium. On the other hand, due to the inherent data heterogeneity across wireless users, training a single model to serve all users can severely degrade individual user performance. Towards addressing this challenge, in this work we proposeover-the-air clustered FL, where multiple models are trained concurrently over-the-air, and each model is adapted gradually to a group of users with similar data distributions. We introduce AirCluster, an over-the-air clustered FL framework with coordinated zero-forcing MIMO beamforming, along with a sketching-based dimensionality reduction mechanism to enable over-the-air training with limited number of antennas. Our theoretical analysis provides formal convergence guarantees for the trained models, while identifying the key performance trade-offs in terms of the convergence rate, compression ratio, channel quality, and the number of antennas. Through extensive experiments on multiple datasets, we observe significant increase in the test accuracy for individual users over state-of-the-art FL benchmarks. Our results demonstrate over-the-air FL to be a promising approach in addressing the communication bottleneck of FL, even under severe data heterogeneity.

Collaborative machine learning enables privacy-preserving training of machine learning models without collecting sensitive client data. Despite recent breakthroughs, communication bottleneck is still a major challenge against its scalability to larger networks. To address this challenge, we propose PICO, the first collaborative learning framework with linear communication complexity, significantly improving over the quadratic state-of-the-art, under formal information-theoretic privacy guarantees. Theoretical analysis demonstrates that PICO slashes the communication cost while achieving equal computational complexity, adversary resilience, robustness to client dropouts, and model accuracy to the state-of-the-art. Extensive experiments demonstrate up to 91x reduction in the communication overhead, and up to 7x speed-up in the wall-clock training time compared to the state-of-the-art. As such, PICO addresses a key technical challenge in multi-party collaborative learning, paving the way for future large-scale privacy-preserving learning frameworks.

Clustered federated learning is a popular paradigm to tackle data heterogeneity in federated learning, by training personalized models for groups of users with similar data distributions. A critical challenge is to protect the privacy of individual user updates, as the latter can reveal extensive information about sensitive local datasets. To do so, a recent promising approach is information-theoretic secure aggregation, where parties learn the aggregate (sum) of user updates, but no further information is revealed about the individual updates. In this work, we present the first secure aggregation frameworks in the context of clustered federated learning, to learn the aggregate of user updates for any clustering of users, but without learning any information about the local updates or cluster identities. Our frameworks can achieve linear communication complexity under formal information-theoretic privacy guarantees, while providing key trade-offs between communication and computation complexity, adversary tolerance, and resilience to user dropouts.

Secure aggregation is a privacy-aware protocol for model aggregation in federated learning. A major challenge of conventional secure aggregation protocols is their large communication overhead. Towards addressing this challenge, in this work we propose the first gradient sparsification framework for communication-efficient secure aggregation, which allows aggregation of sparsified local gradients from a large number of users, without revealing the individual local gradient parameters in the clear. We provide the theoretical performance guarantees of the proposed framework in terms of the communication efficiency, resilience to user dropouts, and model convergence. We further evaluate the performance of our framework through large-scale experiments in a distributed network with up to 100 users, and demonstrate a significant reduction in the communication over-head compared to conventional secure aggregation benchmarks.

Federated learning is a distributed framework for training a machine learning model over the data stored by wireless devices. A major challenge in doing so is the communication overhead from the devices to the server. Over-the-air federated learning is a recent framework to address this challenge, which utilizes the superposition property of the wireless multiple access channel to enable computations to be performed in the wireless medium. Current over-the-air aggregation frameworks, on the other hand, train a single model for all users, which can degrade performance in heterogeneous environments where the data distributions of the users can differ from one another. This work presents a personalized over-the-air federated learning framework towards addressing this challenge. Our experiments demonstrate significant performance improvement in terms of the test accuracy over conventional federated learning.