Privacy-preserving Transformer inference has gained attention due to the potential leakage of private information. Despite recent progress, existing frameworks still fall short of practical model scales, with gaps up to a hundredfold. A possible way to close this gap is the Mixture of Experts (MoE) architecture, which has emerged as a promising technique to scale up model capacity with minimal overhead. However, given that the current secure two-party (2-PC) protocols allow the server to homomorphically compute the FFN layer with its plaintext model weight, under the MoE setting, this could reveal which expert is activated to the server, exposing token-level privacy about the client's input. While naively evaluating all the experts before selection could protect privacy, it nullifies MoE sparsity and incurs the heavy computational overhead that sparse MoE seeks to avoid. To address the privacy and efficiency limitations above, we propose a 2-PC privacy-preserving inference framework, SecMoE. Unifying per-entry circuits in both the MoE layer and piecewise polynomial functions, SecMoE obliviously selects the extracted parameters from circuits and only computes one encrypted entry, which we refer to as Select-Then-Compute. This makes the model for private inference scale to 63× larger while only having a 15.2× increase in end-to-end runtime. Extensive experiments show that, under 5 expert settings, SecMoE lowers the end-to-end private inference communication by 1.8~7.1× and achieves 1.3~3.8× speedup compared to the state-of-the-art (SOTA) protocols.

The fully homomorphic encryption schemes FHEW/TFHE support fast bootstrapping to refresh ciphertexts. However, executing FHEW/TFHE with SIMD bootstrapping is inefficient due to the algebraic structure. For the theoretical state of the art, LW23 (Liu and Wang, Eurocrypt2023) proposed a mathematical framework for SIMD bootstrapping, while the ciphertext slots are limited in size $\tilde O\left( {{\lambda ^{0.25}}} \right)$ and total complexity of $O\left( {{2^{\rho - 1}}n} \right)$ for n(= O(λ)) LWE ciphertexts.This paper aims at developing an efficient FHE scheme with SIMD bootstrapping, called Batch RGSW, which yields overall bootstrapping complexity O(n2/ε), ε > 2. To achieve this, there are two main steps to craft our scheme. Firstly, we decompose a cyclotomic ring into two smaller subrings R1 for the ciphertext space and ℛ2for the base as slots. The ciphertext is then extracted with basic automorphic rotations. In this way, we obtain a batch RGSW scheme with ciphertext slots in size of $\tilde O\left( {{\lambda ^{0.5}}} \right)$. Secondly, we combine Partial Fast Fourier Transformation (PFFT) with Batch RGSW scheme to reduce bootstrapping amortized cost. We utilize PFFT for coefficient decomposition and perform batch homomorphic multiplications. We end up with a bootstrapping algorithm with amortized complexity O(1).