Large language models (LLMs) have seen remarkable growth in recent years. To leverage convenient LLM cloud services, users are inevitably to upload their prompts. Additionally, for tasks such as translation, reading comprehension, and summarization, associated files or context are inherently needed, whether or not they contain user privacy information. Despite the rapid progress in LLM capabilities, research on preserving user privacy during inference has been relatively scarce. To this end, this paper conducts some exploratory research in this domain. Firstly, we show that (1) the embedding space of tokens is highly sparse, and (2) LLMs primarily function in the orthogonal subspace of embedding space, these two factors making privacy extremely vulnerable. Then, we analyze the structural characteristics of LLMs and design a distributed privacy-preserving inference paradigm which can effectively resist privacy attacks. Finally, we perform a thorough evaluation of the defended models on mainstream tasks and find that low-bit quantization techniques can be effectively combined with our inference paradigm, achieving a balance between privacy, utility, and runtime memory efficiency.

Gradient inversion attacks (GIAs) pose significant challenges to the privacy-preserving paradigm of distributed learning. These attacks employ carefully designed strategies to reconstruct victim’s private training data from their shared gradients. However, existing work mainly focuses on attacks and defenses for image-modal data, while the study for text-modal data remains scarce. Furthermore, the performance of the limited attack researches on text-modal data is also unsatisfactory, which can be partially attributed to the finer granularity of text data compared to image. To bridge the existing research gap, we propose a high-fidelity attack method tailored for Transformer-based language models (LMs). In our method, we initially reconstruct the label space of the victim’s training data by leveraging the characteristics of the Transformer architecture. After that, we propose a shallow-to-deep paradigm to facilitate gradient matching, which can significantly improve the attack performance. Furthermore, we develop a weighted surrogate loss that resolves the consistent deviation issue present in current attack researches. A substantial number of experiments on Transformer-based LMs (e.g., Bert and GPT) demonstrate that our attack is competitive and significantly outperforms existing methods. In the final part of this paper, we investigate the influence of the inherent position embedding module within the Transformer architecture on attack performance, and based on the analysis results, we propose a countermeasure to alleviate part of the privacy leakage issue in distributed learning.

Distributed learning frameworks aim to train global models by sharing gradients among clients while preserving the data privacy of each individual client. However, extensive research has demonstrated that these learning frameworks do not absolutely ensure the privacy, as training data can be reconstructed from shared gradients. Nevertheless, the existing privacy-breaking attack methods have certain limitations. Some are applicable only to small models, while others can only recover images in small batch size and low resolutions, or with low fidelity. Furthermore, when there are some data with the same label in a training batch, existing attack methods usually perform poorly. In this work, we successfully address the limitations of existing attacks by two steps. Firstly, we model the coefficient of variation (CV) of features and design an evolutionary algorithm based on the minimum CV to accurately reconstruct the labels of all training data. After that, we propose a stepwise gradient inversion attack, which dynamically adapts the objective function, thereby effectively and rationally promoting the convergence of attack results towards an optimal solution. With these two steps, our method is able to recover high resolution images (224*224 pixel, from ImageNet and Web) with high fidelity in distributed learning scenarios involving complex models and larger batch size. Experiment results demonstrate the superiority of our approach, reveal the potential vulnerabilities of the distributed learning paradigm, and emphasize the necessity of developing more secure mechanisms. Source code is available at https://github.com/MiLab-HITSZ/2023YeHFGradInv.

Deep Neural Networks (DNNs) are widely used for the outstanding performance in many fields. However, the training of DNN models has high requirements for the users’ data and computation resources, so many users with limited resources tend to download pre-trained models from some platforms and then finetune the pre-trained models to match their own tasks. However, the pre-trained models are under the threat of the backdoor attack. The backdoor attackers inject backdoors in the models, leading the backdoor models to predict target predictions designed by the attackers in advance. However, most existing backdoor model inspection methods rely on the clean data samples from the dataset of the model, which are difficult to get for users who just download the pre-trained models from the platforms. There are also a few defense methods not dependent on the data, but they also have their limits in practice. We propose Data-Free Masking and Reverse Engineering Loops (DF-MREL), a simple yet efficient data-free method for backdoor model inspection, which is widely applicable when resources are limited. Our experiments show its excellent performance in detecting backdoor models. Source code will be published after accepted.