With the development of cryptographic tools such as Fully Homomorphic Encryption (FHE) and secure Multiparty Computation (MPC), privacy-preserving Machine Learning as a Service (MLaaS) has gained attractiveness for its security when it comes to utilizing cross-domain data. However, cryptographic tools are characterized by huge overhead, which results in the MLaaS quality being unbearably degraded, especially for latency-sensitive MLaaS applications. In this paper, we focus on the problem of low-latency inference associated with MLaaS and propose CrossNet, a Privacy-preserving Neural Network Inference (PPNI) framework based on FHE, for applications with limited client-side computational and communication resources. CrossNet performs model transformations on neural networks so that they can be evaluated in an FHE-friendly manner. Model transformation introduces limited interactions between client and server, thus restricting inference latency. In addition, CrossNet includes a series of layer constructions where elaborate encoding forms and computational orders are designed to further reduce the overhead of transformed layers. CrossNet outperforms the existing FHE-based frameworks by 4x efficiency and reduces nearly 30% inference latency on ResNet-50 in a resource-limited setting.