Fine-tuning large language models (LLMs) on down-stream tasks has become a standard approach to adapt their capabilities. However, the process raises privacy concerns when using sensitive datasets, prompting increasing interest in differentially private (DP) fine-tuning methods. While existing approaches build upon the seminal work of DP-SGD, they are constrained by the inherent inefficiency bottlenecks. In this paper, we investigate the potential of DP zeroth-order methods for LLM fine-tuning, which avoids the scalability bottleneck of SGD by approximating gradients with more efficient zeroth-order gradients. We propose the stagewise DP zeroth-order method (DP-ZOSO) that dynamically schedules key hyperparameters to leverage the synergy between DP random perturbation and the gradient approximation error. To further enhance the scalability, we propose DP zeroth-order stagewise pruning method (DP-ZOPO) which reduces the trainable parameters by a data-free pruning technique requiring no extra privacy budget. We provide theoretical analysis for both proposed methods and conduct extensive empirical analysis on both encoder-only masked and decoder-only autoregressive language model, achieving impressive results in terms of scalability and utility across diverse tasks.

Nowadays, there exists a lot of cross-region data transmission demand on the cloud. It is promising to use serverless computing for data compressing to save the total data size. However, it is challenging to estimate the data transmission time and monetary cost with serverless compression. In addition, minimizing the data transmission cost is non-trivial due to the enormous parameter space. This paper focuses on this problem and makes the following contributions: 1) We propose empirical data transmission time and monetary cost models based on serverless compression. It can also predict compression information, e.g., ratio and speed using chunk sampling and machine learning techniques. 2) For single-task cloud data transmission, we propose two efficient parameter search methods based on Sequential Quadratic Programming (SQP) and Eliminate then Divide and Conquer (EDC) with proven error upper bounds. Besides, we propose a parameter fine-tuning strategy to deal with transmission bandwidth variance. 3) Furthermore, for multi-task scenarios, a parameter search method based on dynamic programming and numerical computation is proposed. We have implemented the system called Fluid-Shuttle, which includes straggler optimization, cache optimization, and the autoscaling decompression mechanism. Finally, we evaluate the performance of Fluid-Shuttle with various workloads and applications on the real-world AWS serverless computing platform. Experimental results show that the proposed approach can improve the parameter search efficiency by over$3\times $compared with the state-of-art methods and achieves better parameter quality. In addition, our approach achieves higher time efficiency and lower monetary cost compared with competing cloud data transmission approaches.