The recommendation system is an effective approach to alleviate the information overload caused by the popularization of the Internet. Existing recommendation methods often use advanced deep learning algorithms to predict user preferences. The diffusion model is a deep generative model that has received much attention in recent years and has been successfully applied in recommendation systems. However, previous research has mainly used MLP in the reverse process of the diffusion model, which fails to fully utilize the collective signals of various items in the recommendation system. This article improves the diffusion recommendation model by introducing a carefully designed graph-based decoding network (GDN) in the reverse process. GDN improves recommendation performance by introducing relationships between items via the item-item graph. In addition, skip connections and normalization layers are implemented to maintain low-order neighbor information. Experiments are conducted to compare the proposed model with several state-of-the-art recommendation methods on three real-world datasets, which demonstrate the improvement of the proposed method over the diffusion recommendation model. Specifically, the proposed method outperforms the diffusion recommendation model with autoencoder (AE) by 21.67% on average. The contribution of each component of the proposed model is also illustrated by the ablation experiments. The implementation codes of the proposed model are available via https://github.com/crx1729/G-Diff.