We present the first digital twin framework that operationalizes the production of multi-decadal, global climate projections at kilometre-scale resolution, developed within the European Union’s Destination Earth initiative. Using three coupled Earth system models and selected impact-sector applications, we have built end-to-end workflows for both regular and on-demand climate projections on two EuroHPC supercomputers, LUMI and MareNostrum5. These workflows produced the first-ever multi-decadal simulations at 5 km resolution across all major Earth system components, using the same output parameters and grid, and achieving a production throughput of 0.6 simulated years per day and a climate data portfolio of 6.6 petabytes. We demonstrate the scalability of two of these Earth system models across both CPU and GPU-based systems at global resolutions up to 1 km, across atmosphere, ocean, land, and sea-ice, and report record-breaking full-machine performance on LUMI and MareNostrum5 of up to 97 simulated days per day at 1 km resolution.

In this paper we present a framework to enable data-intensive Spark workloads on MareNostrum, a petascale supercomputer designed mainly for compute-intensive applications. As far as we know, this is the first attempt to investigate optimized deployment configurations of Spark on a petascale HPC setup. We detail the design of the framework and present some benchmark data to provide insights into the scalabilityof the system. We examine the impact of different configurations including parallelism, storage and networking alternatives, and we discuss several aspects in executing Big Data workloads on a computing system that is based on the compute-centric paradigm. Further, we derive conclusions aiming to pave the way towards systematic and optimized methodologies for fine-tuning data-intensive application on large clusters emphasizing on parallelism configurations.