There has been growing interest in improving the BERT inference throughput on resource-constrained edge devices for a satisfactory user experience. One methodology is to employ heterogeneous computing, which utilizes multiple processing elements to accelerate inference. Another methodology is to deploy Neural Architecture Search (NAS) to find optimal solutions in accuracy-throughput design space. In this paper, for the first time, we incorporate NAS with pipelining for BERT models. We show that performing NAS with pipelining achieves on average 53% higher throughput, compared to NAS with a homogeneous system.

A retrospective view is presented of the Charlotte distributed operating system, a testbed for developing techniques and tools to solve computation-intensive problems with large-grain parallelism. The final version of Charlotte runs on the Crystal multicomputer, a collection of VAX-11/750 computers connected by a local area network. The kernel/process interface is unique in its support for symmetric, bidirectional communication paths (called links), and synchronous nonblocking communications. Several lessons were learned in implementing Charlotte. Links have proven to be a useful abstraction, but the primitives do not seem to be at quite the right level of abstraction. The implementation uses finite-state machines and a multitask kernel, both of which work well. It also maintains absolute distributed information which is more expensive that using hints. The development of high-level tools, particularly the Lynx distributed programming language, has simplified the use of kernal primitives and helps to manage concurrency at the process level.>