Efforts to mitigate lock contention from concurrent threaded accesses to MPI have reduced contention through fine-grained locking, avoided locking altogether by offloading communication to dedicated threads, or alleviated negative side effects from contention by using better lock management protocols. The blocking nature of lock-based methods, however, wastes the asynchrony benefits of nonblocking MPI operations, and the offloading model sacrifices CPU resources and incurs unnecessary software offloading overheads under low contention.

This paper provides an in-depth analysis of the software overheads in the MPI performance-critical path and exposes mandatory performance overheads that are unavoidable based on the MPI-3.1 specification. We first present a highly optimized implementation of the MPI-3.1 standard in which the communication stack---all the way from the application to the low-level network communication API---takes only a few tens of instructions. We carefully study these instructions and analyze the root cause of the overheads based on specific requirements from the MPI standard that are unavoidable under the current MPI standard. We recommend potential changes to the MPI standard that can minimize these overheads. Our experimental results on a variety of network architectures and applications demonstrate significant benefits from our proposed changes.

PARSEC is a massively parallel Density-Functional-Theory (DFT) code. Within the modernization effort towards the new Intel Knights Landing platform, we adapted the main computational kernel, represented as high-order finite-difference stencils, to use hybrid MPI and OpenMP* runtimes. We also employed MPI-3 non-blocking neighborhood collectives for the halo exchange. We present performance data on the Knights Landing platform including MPI traces portraying our exploration of communication-computation overlap in a hybrid MPI and OpenMP application, and the fine-tuning of the interplay between the load-balancing - static at the MPI level and dynamic in OpenMP.