Model-heterogeneous federated tuning (MHFT) enables the privacy-preserving fine-tuning of foundation models in heterogeneous systems by allowing clients and the server to adopt different model architectures. Depth partial training—where each client updates only a subset of the model's layers—alleviates system heterogeneity but exacerbates client drift, which stems from clients optimizing different objectives and therefore degrades overall performance. Beyond the well-known statistical bias—where non-IID data leads to client drift—we identify a structural bias arising from clients deploying only partial layers of the global model, which serves as an important cause of drift. We further provide a theoretical analysis showing that the possible range of structural bias expands linearly with the number of missing layers. To counter this effect, we introduce FedBRICK (Federated Bias Recovery via Inserted Calibrative Kernels), which inserts tiny BRICKs into each client’s subnetwork. We employ a dual-end layer-wise distillation scheme to train these blocks using both client-side local data and a small public proxy set on the server. This design effectively mitigates the structural bias caused by layer dropping, reduces client drift, and remains practical for storage-constrained devices. Extensive experiments on federated learning benchmarks confirm that FedBRICK delivers up to a 5% average accuracy gain while requiring no more than 1.44% extra storage per client.

Large Language Models (LLMs) exhibit remarkable language understanding and problem-solving capabilities but are heavily reliant on vast amounts of high-quality data, which raises significant privacy concerns. Federated Learning (FL) presents a decentralized training paradigm that minimizes raw data sharing but is still vulnerable to privacy attacks that exploit shared model updates. Federated LLMs, especially dialogue models, are particularly prone to query-based attacks, which can lead to the unintentional disclosure of memorized private information. This study performs extensive experiments across multiple model sizes and federated architectures to assess the extent of query-based privacy leakage and evaluate existing protection mechanisms. To enable standardized and reproducible privacy assessments, we propose a multi-stage privacy leakage detection method, integrating structured privacy attribute extraction, controlled injection, realistic prompting, and quantitative evaluation under both generalized and personalized privacy scenarios. Additionally, we introduce FedShieldLLM, a lightweight, adaptable client-side anonymization framework that performs personalized data de-identification prior to federated training, effectively mitigating privacy risks while preserving task utility. Our contributions include: (1) a systematic analysis of query-based privacy leakage in federated LLMs; (2) the design of a reproducible, architecture-agnostic multi-stage detection method; and (3) the development of FedShieldLLM for secure and flexible client participation in privacy-sensitive federated learning.

Federated learning (FL) stands out as a promising solution to address the ever-increasing data scarcity problem of large language model (LLM) training through collecting data from distributed sources. The unique challenges however exist in three aspects, the extreme large communication overhead due to frequent model aggregation, the suboptimal performance due to data heterogeneity, and privacy leakage risk due to model parameter exchange. To address these issues, we propose Federated Knowledge Learning (FKL), a novel token-level knowledge sharing framework tailored for LLM to achieve effective task relevant knowledge aggregation. FKL offers the following advantages: (1) Adaptive Flexibility: Clients and servers can employ models customized for their specific tasks while ensuring seamless collaboration. Each node within the system functions independently and can be flexibly adapted as needed. (2) Communication Efficiency: Instead of huge model or parameter exchange, the unique knowledge sharing mechanism of FKL can significantly reduce the communication overhead while delivering superior performance. (3) Privacy Safeguards: The distributed token-level knowledge-sharing mechanism ensures privacy by deidentifying sensitive information and mitigating potential leakage risks. Experiments on LLMs (125M-7B parameters) validate FKL's versatility, achieving 50% higher accuracy on GLUE benchmarks and a notable 0.7x improvement in privacy task response quality compared to existing FL-based LLM approaches.

In recent years, the distributed training of foundation models (FMs) has seen a surge in popularity. In particular, federated learning enables collaborative model training among edge clients while safeguarding the privacy of their data. However, federated training of FMs across resource-constrained and highly heterogeneous edge devices encounter several challenges. These include the difficulty of deploying FMs on clients with limited computational resources and the high computation and communication costs associated with fine-tuning and collaborative training. To address these challenges, we propose FedCKMS, a Cluster-Aware Framework with Knowledge-Aware Model Search. Specifically, FedCKMS incorporates three key components. The first component is multi-factor heterogeneity-aware clustering, which groups clients based on both data distribution and resource limitations and selects an appropriate model for each cluster. The second component is knowledge-aware model architecture search, which enables each client to identify the optimal sub-model from the cluster model, facilitating adaptive deployment that accommodates highly heterogeneous computational resources across clients. The final component is cluster-aware knowledge transfer, which facilitates knowledge sharing between clusters and the server, addressing model heterogeneity, and reducing communication overhead. Extensive experiments demonstrate that FedCKMS outperforms state-of-the-art baselines by 3-10% in accuracy.

Many networking tasks now employ deep learning (DL) to solve complex prediction and optimization problems. However, current design philosophy of DL-based algorithms entails intensive engineering overhead due to the manual design of deep neural networks (DNNs) for different networking tasks. Besides, DNNs tend to achieve poor generalization performance on unseen data distributions/environments.