Federated deep learning (FDL) is a promising privacy-preserving approach for training deep neural networks on distributed datasets without raw data sharing. But the classical synchronous FDL faces straggler problem: slow trainers severely impede overall efficiency. Inspired by speculative execution techniques in modern processors, this paper proposes SwiftFL, a novel and efficient speculative training system for FDL. Instead of simply waiting for slower trainer, SwiftFL proactively updates the global model with predicted gradients, enabling faster trainers to speculatively initiate the next training round. Furthermore, a gradient compensation technique is proposed to correct mispredicted training without re-training. Finally, to overcome the model-drift problem caused by fast trainers perform more local training rounds, we propose a client selection strategy. This strategy determines whether trainers should perform speculative training by striking a balance between two metrics: model drift degree and local training efficiency. In the evaluation, we compare SwiftFL with four state-of-the-art FDL systems and demonstrate that SwiftFL achieves an average speedup of 6.08× while maintaining consistent final model accuracy.

Cloud AI services have experienced rapid development, raising concerns to privacy protection of cloud tenants. Many proposals have been made to use the NPU Trusted Execution Environment (TEE) to protect AI workloads on the cloud. However, existing designs typically bind NPUs exclusively to a single tenant, preventing multiple tenants from sharing the computing power of the TEE-NPUs.As such, we have designed a novel TEE for discrete NPUs, named NXT (NPU eXtension for Trust), which breaks the exclusive binding architecture and allows multiple tenants to securely share the TEE-NPU cluster. Firstly, we introduced an NPU Trusted Agent (NTA) to the most privileged level of the TEE system to assist in the global scheduling of the TEE-NPU cluster. Secondly, we implemented a flexible isolation mechanism to provide security for multi-tenant fine-grained sharing of NPU resources. Thirdly, we support efficient communication between workloads within TEE-NPUs and with legacy NPUs, accelerating multi-workload collaborative computing. We evaluated NXT by extending gem5 and a cycle-accurate NPU simulator to build a prototype. Results show that NXT improves overall utilization by up to 3.49× and ANTT by up to 7.24×, with only 6.38% average overhead for scheduling and isolation. It also boosts parallel inference performance by 58.3% for GPT-2(XL) and 63.6% for LLaMA-13B compared to static protection schemes.

With the growing use of large language models (LLMs) hosted on cloud platforms to offer inference services, privacy concerns about the potential leakage of sensitive information are escalating. Secure Multi-Party Computation (MPC) is a promising solution to protect the privacy in LLM inference. However, MPC requires frequent inter-server communication, causing high performance overhead. Inspired by the prevalent activation sparsity of LLMs, where most neuron are not activated after non-linear activation functions, we propose an efficient private inference system, Comet. This system employs an accurate and fast predictor to predict the sparsity distribution of activation function output. Additionally, we introduce a new private inference protocol. It efficiently and securely avoids computations involving zero values by exploiting the spatial locality of the predicted sparsity distribution. While this computation-avoidance approach impacts the spatiotemporal continuity of KV cache entries, we address this challenge with a low-communication overhead cache refilling strategy that merges miss requests and incorporates a prefetching mechanism. Finally, we evaluate Comet on four common LLMs and compare it with six state-of-the-art private inference systems. Comet achieves a$1.87\times-2.63\times$speedup and a$1.94\times-2.64\times$communication reduction.