Existing road damage detection methods typically rely on a single unified model trained under standard conditions to extract damage features and model spatial relationships between damage regions and the surrounding background. However, road damage exhibits highly diverse relational patterns due to variations in morphology, scale, and contextual appearance, making it difficult for a single model to effectively capture such heterogeneity, especially under complex weather and illumination conditions. To overcome this limitation, we propose a mixture-of-experts–based road damage detection framework that decomposes complex relational modeling into multiple specialized expert processes. A shallow detail-perceptive mixture-of-experts (SDP-MoE) module is introduced to enhance the extraction of fine-grained texture and structural cues critical for accurate damage localization. Meanwhile, a mixture-of-experts gated cross-scale attention (MEGCSA) module is designed to model heterogeneous contextual relationships across multi-scale features, enabling effective integration of local details and global semantics. By collaboratively leveraging specialized experts, the proposed framework provides a flexible and expressive mechanism for multi-scale and multi-type relational modeling in road damage detection. In addition, expert-oriented data augmentation and a load-balancing loss are employed to promote stable and balanced expert learning. Extensive experiments on the CNRDD, RDD2022, and ARSDD benchmarks under challenging low-light and rainy conditions demonstrate that the proposed method consistently outperforms the baseline, achieving m A P @ 0 . 5 improvements of 4.5%, 6.6%, and 4.2%, respectively. More importantly, robustness analysis shows substantially reduced performance degradation in complex road scenarios compared with single-model detectors. Furthermore, the modular and plug-and-play design enables seamless integration into existing road damage detection systems, highlighting its strong practical applicability.