Electro-hydraulic servo valve is a complex component integrating machine, electricity, and fluid, which is widely used in aerospace hydraulic system. It is a key component of the hydraulic system, and as a highly reliable and integrated component, faults are often concealed, and acquiring labeled fault samples is challenging. These factors limit the development of efficient fault diagnose based method of data-driven. In this paper, a hybrid digital twin modeling technique combining physical model and data-driven is proposed for electro-hydraulic servo valve fault diagnosis under insufficient or uneven sample size. Firstly, a high-fidelity digital twin model of the servo valve is built by combining virtual simulation based on physical model and generative adversarial network. Then using the built digital twin model, simulated signals under fault conditions are generated to expand the sample size and train the data-driven convolutional neural network-based fault diagnosis model. The experimental results show that the proposed diagnostic framework can solve the problem of the lack of sample size of the hydraulic system and effectively improve the accuracy of fault diagnosis. The proposed combined physical and data-driven digital twin framework can be applied to other hydraulic systems and fields..