Large Language Models (LLMs) exhibit remarkable language understanding and problem-solving capabilities but are heavily reliant on vast amounts of high-quality data, which raises significant privacy concerns. Federated Learning (FL) presents a decentralized training paradigm that minimizes raw data sharing but is still vulnerable to privacy attacks that exploit shared model updates. Federated LLMs, especially dialogue models, are particularly prone to query-based attacks, which can lead to the unintentional disclosure of memorized private information. This study performs extensive experiments across multiple model sizes and federated architectures to assess the extent of query-based privacy leakage and evaluate existing protection mechanisms. To enable standardized and reproducible privacy assessments, we propose a multi-stage privacy leakage detection method, integrating structured privacy attribute extraction, controlled injection, realistic prompting, and quantitative evaluation under both generalized and personalized privacy scenarios. Additionally, we introduce FedShieldLLM, a lightweight, adaptable client-side anonymization framework that performs personalized data de-identification prior to federated training, effectively mitigating privacy risks while preserving task utility. Our contributions include: (1) a systematic analysis of query-based privacy leakage in federated LLMs; (2) the design of a reproducible, architecture-agnostic multi-stage detection method; and (3) the development of FedShieldLLM for secure and flexible client participation in privacy-sensitive federated learning.

Federated learning (FL) stands out as a promising solution to address the ever-increasing data scarcity problem of large language model (LLM) training through collecting data from distributed sources. The unique challenges however exist in three aspects, the extreme large communication overhead due to frequent model aggregation, the suboptimal performance due to data heterogeneity, and privacy leakage risk due to model parameter exchange. To address these issues, we propose Federated Knowledge Learning (FKL), a novel token-level knowledge sharing framework tailored for LLM to achieve effective task relevant knowledge aggregation. FKL offers the following advantages: (1) Adaptive Flexibility: Clients and servers can employ models customized for their specific tasks while ensuring seamless collaboration. Each node within the system functions independently and can be flexibly adapted as needed. (2) Communication Efficiency: Instead of huge model or parameter exchange, the unique knowledge sharing mechanism of FKL can significantly reduce the communication overhead while delivering superior performance. (3) Privacy Safeguards: The distributed token-level knowledge-sharing mechanism ensures privacy by deidentifying sensitive information and mitigating potential leakage risks. Experiments on LLMs (125M-7B parameters) validate FKL's versatility, achieving 50% higher accuracy on GLUE benchmarks and a notable 0.7x improvement in privacy task response quality compared to existing FL-based LLM approaches.