The Transformer model, which has significantly advanced natural language processing and computer vision, overcomes the limitations of recurrent neural networks and convolutional neural networks. However, it faces challenges with computational efficiency and memory management due to complex computations and variable-length inputs. Despite research efforts, these issues persist. This paper presents a novel optimization of the Transformer model, following an indepth analysis of its computational graph structure. Firstly, we utilize Deep Computing Unit (DCU) as our hardware platform. Secondly, we optimize element-wise and reduction operators through operator fusion and rewriting. Thirdly, we develop a fine-grained memory management algorithm using a greedy strategy. As a result, the training speed of the Transformer model increases by 1.2x - 1.4x without compromising accuracy.