To satisfy the growing demand for traffic prediction induced by urbanization, the intelligent transportation system integrated various cutting-edge artificial intelligence technologies, with large language models (LLMs) as a representative, has been developed. However, existing methods are mostly confined by shallow LLMs utilization, where the semantic capacity of LLMs is ignored and the traffic data are directly fed in. Furthermore, the modality diversity of different traffic prediction scenarios (e.g., flow, speed, and demanding) remains to be underexplored, which restricts the model flexibility towards downstream applications. To mitigate these limitations, we propose a Mixture of Semantic and Spatial Experts (SS-MoE) for traffic prediction along with the human-intelligible post-hoc result explanation. Specifically, to enlighten the traffic predictor with abundant semantic information, we design hierarchically coarse- and fine-grained prompts including role assignments, dataset descriptions, and background supplements, which serves as the auxiliary knowledge for downstream prediction. Afterwards, considering the diversity of real-world traffic scenarios, we construct the MoE framework consisting of a spatial expert, a semantic expert, and a general expert, which accounts for the node-level features, the semantic representations, and the overall generalization, respectively. At last, we instruct the LLM to explain and analyze the final prediction, which is able to provide insightful conclusions and support intelligent transportation decisions, forming a unified prediction-explanation pipeline. Extensive experiments on five public traffic datasets demonstrate the superiority of SS-MoE across three traffic prediction tasks. Experimental results indicate that the MAE and RMSE values of SS-MoE are reduced by up to 4.04% and 3.20% compared with that of the runner-up, respectively.