Behavior biometrics-based user authentication with Wi-Fi gains significant attention due to its ubiquitous and contact-free manners. An individual’s identity can be verified by analyzing activities induced signal variances, excellently balancing the security demands and user experience. However, the inherent complexity of Wi-Fi signals presents significant challenges for behavior biometrics-based user authentication. The susceptibility of Wi-Fi signals results in a poor cross-environment generalization capability, which is overlooked by the existing research. In addition, most existing works of behavior-based user authentication are based on one-off activity. This makes them vulnerable to zero-effort attacks and imitation attacks. To address these issues, we propose a cross-environment continuous gesture-based user authentication framework with Wi-Fi, dubbed Wi-CGAuth. Specifically, the cross-environment generalization capability is enhanced by the cross-layer joint optimization approach. At the lowest signal layer, the signals’ time, spatial, and frequency diversity are extended maximally, by a novel, subcarrier-level, cost-effective signal optimization strategy. At the middle layer, the multi-view fusion method, i.e., multi-transfer component analysis (TCA), is applied to refine the signals from transceiver pairs after signal preprocessing. The continuous gesture segmentation problem is modeled as the classification problem, which is solved by CNN. At the upper layer, a Convolutional Neural Network-Transformer (CNN-Transformer) model is employed to achieve the dual task of effective user authentication and accurate gesture recognition. After extensive experiments in three typical indoor scenarios, Wi-CGAuth can achieve an average authentication accuracy of 92.7%, demonstrating its robustness and effectiveness.