Current advances in processor technology, and the rapid development of high speed networking technology, such as ATM, have made high performance network computing an attractive computing environment for large-scale high performance distributed computing (HPDC) applications. However, due to the communications overhead at the host-network interface, most of the HPDC applications are not getting the full benefit of high speed communication networks. This overhead can be attributed to the high cost of operating system calls, context switching, the use of inefficient communication protocols, and the coupling of data and control paths. We present an architecture and implementation for a low-latency, high-throughput message passing tool, that we refer to as the NYNET (ATM wide area network testbed in New York state) Communication System (NCS), which can support a variety of HPDC applications with different Quality of Services (QOS) requirements. NCS uses multithreading to provide efficient techniques that overlap computation and communication. NCS uses read/write trap routines to bypass traditional operating system calls. This reduces latency and avoids using inefficient communication protocols. By separating data and control paths, NCS eliminates unnecessary control transfers. This optimizes the data path and improves performance. Benchmarking results show that the performance of NCS is at least a factor of two better than the performance of corresponding p4 and PVM primitives.