Accurate spatio-temporal traffic prediction is essential for optimizing urban traffic management and resource allocation. To reduce the cost and complexity of cross-city deployment, recent studies have explored spatio-temporal foundation models capable of accurate zero-shot prediction. However, these models are limited to single-modal data, which restricts their capacity to capture the complexity of real-world traffic dynamics. The increasing availability of multi-modality data—such as satellite imagery and points of interest (POI)—offers a promising avenue for enhancing cross-city traffic prediction by providing richer background contexts. Despite this potential, developing foundational models for multi-modality spatio-temporal prediction presents two challenges: the availability and quality of multi-modality data vary significantly across cities, with some cities lacking certain modalities or containing noisy information; and spatial patterns are highly localized and specific to individual regions, which hinders generalization. To address these challenges, we propose MoST, a foundation model for multi-modality spatio-temporal traffic prediction. We introduce a Multi-modality Refinement Module that encodes available modality data and adaptively selects task-relevant modalities while suppressing noisy modalities. Furthermore, we design a Spatio-Temporal Prediction Module that incorporates a spatial expert selection mechanism guided by multi-modality cues. This mechanism dynamically identifies region-specific spatial patterns and assigns appropriate spatial experts to model local dependencies. Finally, we conduct extensive experiments on real-world datasets to validate the superior performance and strong generalization capability of MoST.

Large models are evolving towards massive scale, diverse model architectures (dense and sparse) and long-context processing, which makes it very challenging to efficiently scale large models on parallel machines. The current widely-used parallelization strategies are often sub-optimal due to their limited parallelization strategy space. To this end, we propose Hypertron, a scalable parallel large-model training framework which incorporates an unprecedented high-dimensional (up to 7D) parallelization space, a holistic scheme for efficient dimension fusion, and a comprehensive performance model to guide the high-dimensional exploration. By exploiting the high-dimensional space to discover the optimal strategy which is not supported by existing frameworks, Hypertron significantly reduces memory and communication cost while improving parallel scalability. Extensive evaluations demonstrate that Hypertron achieves up to 56.7% Model FLOPs Utilization (MFU) on 2,048 new-generation Ascend NPU accelerators (scaling with supernodes) for different large models (such as sparse 141B and dense 310B), with 1.33x speedup over the best configuration of the state-of-the-art frameworks.