Positron Emission Tomography (PET) is a critical imaging modality in nuclear medicine but requires radioactive tracer administration, which increases radiation exposure risks. While recent studies have investigated MR-guided low-dose PET denoising, they neglect two critical factors: the synergistic roles of multicontrast MR images and disease-specific denoising requirements. In this work, we propose a diffusion model that integrates T1-weighted, T2 fluid attenuated inversion recovery (T2 FLAIR), and hippocampal-optimized (T2 HIPPO) MR sequences to achieve ultra-low-dose PET denoising tailored for temporal lobe epilepsy (TLE). Our parallel cross-modal fusion (PCMF) module employs dedicated encoders to extract cross-modal features-which are dynamically integrated via attention mechanisms. Extensive experiments demonstrate that our method outperforms other approaches in preserving image quality. The PSNR and SSIM obtained were 37.0251 $\pm$ 1.5215 dB and 0.9760 $\pm$ 0.0057 (p < 0.01). Compared to the PET-only baseline model (IDDPM), our method achieved improvements of 8.4% in PSNR and 1.7% in SSIM, particularly excelling in diagnostically relevant temporal and hippocampal regions. This method provides a novel pathway for disease-specific PET denoising and has the potential to be generalized to other neurodegenerative diseases.