Face morphing attacks pose a significant threat to society as they disrupt the one-to-one mapping between facial images and identity features in face recognition systems. Despite the development of several detection methods to counter such attacks, the task of restoring the facial image of the accomplice from the morphed facial image, known as face de-morphing, remains a challenging problem. In this paper, we propose a novel diffusion-based method for face de-morphing. This method employs pre-trained diffusion autoencoders to encode the image into two subspaces: a semantic latent space that captures identity features and a stochastic latent space that retains the remaining stochastic details. To ensure the effective separation of identity features, a dual-branch identity separation network is constructed in the semantic latent space. This network utilizes a cross-attention inverse linear interpolation branch to separate the accomplice’s semantic latent code and a multilayer perceptron branch to complement the separated latent code. Additionally, the morphed stochastic latent code is empirically chosen as the accomplice’s stochastic latent code. Finally, a conditional denoising diffusion implicit model is used to decode the latent code of the two subspaces, thus achieving the restoration of the accomplice’s facial image. Experimental results and analysis demonstrate that the proposed method outperforms existing face de-morphing methods in terms of restoration accuracy and image quality.