Ensemble Kalman filter (EnKF) is one of the most important methods for data assimilation, which is widely applied to the reconstruction of observed historical data for providing initial conditions of numerical atmospheric and oceanic models. With the improvement of data resolution and the increase in the amount of model data, the scalability of recent parallel implementations suffers from high overhead on data transfer. In this paper, we propose, S-EnKF: a scalable and distributed EnKF adaptation for modern clusters. With an in-depth analysis of new requirements brought forward by recent frameworks and limitations of current designs, we present a co-design of S-EnKF. For fully exploiting the resources available in modern parallel file systems, we design a concurrent access approach to accelerate the process of reading large amounts of background data. Through a deeper investigation of the data dependence relations, we modify EnKF's workflow to maximize the overlap of file reading and local analysis with a new multi-stage computation approach. Furthermore, we push the envelope of performance further with aggressive co-design of auto-tuning through tradeoff between the benefit on runtime and the cost on processors based on classic cost models. The experimental evaluation of S-EnKF demonstrates nearly ideal strong scalability on up to 12,000 processors. The largest run sustains a performance of 3x-speedup compared with P-EnKF, which represents the state-of-art parallel implementation of EnKF.

The industrial information grid is a special kind of system, the users of which exclusively own geographically distributed computing resources for business service, and try to maintain the lowest total cost of ownership while guaranteeing quality of service. In this paper, we classify the industrial information grid as an extension to grid problem; develop a unified management framework for new management paradigm, which supports the distribution of administration labor and collaboration of system administrator at different locations; propose a self-organizing algorithm, which supports the initial establishment, daily management and exception processing of industrial information grid. Finally, we evaluate the performance of system management, and analyze the management overhead with this new management paradigm.