With the trend of lightweight in the field of intelligent electric vehicles and 3C, the demand for high precision machining of aluminum alloy parts is growing. And tool condition monitoring (TCM) is very important for quality control of parts, so intelligent high-accuracy wear prediction of aluminum alloy high precision machining tools has great industrial application value at present and in the future. This paper presents a novel TCM model (Conv-PhyFormer) of Transformer with physics informed. The model has excellent ability to capture short-term and long-term dependencies from nonlinear cutting time series data when there are few training samples. The embedded hard physical constraint and soft physical constraint in the model make the model partially interpretable. Soft physical constraint in the form of one-dimensional causal convolution can help the proposed model better learn the local context. Hard physical constraint in the form of the mathematical equation representing cutting physical knowledge are embedded, thus the model does not need to learn this knowledge from time series data from scratch. A large number of analysis results of aluminum alloy machining experimental data show that the proposed Conv-PhyFormer has significantly superior prediction accuracy and robustness compared with the current three popular deep learning models for TCM. Embedded soft and hard physical constraints can significantly reduce the training epochs of Transformer prediction model.