Image deraining typically involves synthesizing low-quality degraded data for training using a predefined degraded model of a single weather condition. While in real world scenarios, varying rain intensities result in different sizes and densities of raindrops and rain streaks, increasing the complexity of image degradation. In this paper, we proposed a multi-stage deraining framework based on pre-trained diffusion model, it can efficiently perform the rain removal task under a variety of weather situations. We diffuse degraded images into a noisy state where various types of degradation are transformed into Gaussian noise. Then, during the denoising process, the low-frequency information of the image is replaced through iterative refinement, guiding the pre-trained diffusion model for image reconstruction. Our method effectively utilizes the generative priors in diffusion models and avoid the computational burden of retraining conditional diffusion models. Experimental results on four rainy degradation image datasets show its robustness to different types and severities of degradation (such as raindrops and rain streaks). Compared to recent deraining algorithms, our method achieves a maximum improvement of 0.96 dB (3.5%) in PSNR and 0.021 dB (2.7%) in SSIM for the restored images.