Multi-source image fusion combines infrared and visible information to improve scene perception in applications such as drone reconnaissance and autonomous driving. However, most existing infrared-visible image fusion methods are developed under ideal imaging assumptions. In adverse environments, visible images often lose structural and textural details, whereas infrared images are affected by noise, stripe artifacts, and low contrast, leading to degraded fusion quality and weakened downstream perception performance. To address these limitations, we propose a unified Degradation-aware Restoration and Detail-preserving Fusion Network (DRDFNet), which consists of a Degradation-Aware Restoration Transformer and a Detail-Preserving Fusion Mamba. The restoration branch uses a Compound Degradation Restoration Module (CDRM) to remove complex degradations, while the fusion branch employs a Dynamic Feature Fusion Module (DFFM) to integrate local complementary cues and global correlations across modalities. A two-stage training strategy is further introduced to reduce the optimization conflict between restoration and fusion. In addition, we construct DIVIF, a large-scale degraded IVIF benchmark generated by a physics-based imaging simulator. Experiments on the DIVIF and AWMM-100k benchmarks demonstrate that DRDFNet achieves robust and competitive performance compared with SOTA methods. Both the dataset and source code will be made publicly available at https://github.com/Liupeng97/DRDFNet.