To maximize the cost-effectiveness of neural network (NN) accelerators, architects are actively developing single-chip accelerators which can execute many NNs simultaneously. However, previous approaches fail to achieve full performance potential by exploiting only spatial or temporal resource sharing (SS or TS). They also do not consider memory management that can significantly affect performance. This limitation leads to the dire need for a new multi-NN accelerator taking both opportunities with careful memory management. But, it is extremely challenging to design an ideal spatio-temporal sharing accelerator because it requires (1) an algorithm that determines the degree of SS/TS in large exploration spaces, (2) a new STS-enabled accelerator devised with diverse design points, and (3) carefully-designed memory management that minimizes resource contention during numerous data transfers upon reconfiguration. To this end, we propose STfusion, a fast and flexible multi-NN execution architecture. First, STfusion partitions an accelerator into multiple smaller TS-enabled accelerators. Second, STfusion dynamically fuses small accelerators to adjust the accelerator sizes. Third, STfusion manages on-chip buffer in a page-granularity for stall-free data transfers. Lastly, STfusion provides an algorithm that determines the degree of SS/TS to achieve high throughput while satisfying QoS goals. Our evaluation shows that STfusion significantly outperforms state-of-the-art multi-NN accelerators.

Emerging natural language processing (NLP) models have become more complex and bigger to provide more sophisticated NLP services. Accordingly, there is also a strong demand for scalable and flexible computer infrastructure to support these large-scale, complex, and diverse NLP models. However, existing proposals cannot provide enough scalability and flexibility as they neither identify nor optimize a wide spectrum of performance-critical operations appearing in recent NLP models and only focus on optimizing specific operations. In this paper, we propose NLP-Fast, a novel system solution to accelerate a wide spectrum of large-scale NLP models. NLP-Fast mainly consists of two parts: (1) NLP-Perf: an in-depth performance analysis tool to identify critical operations in emerging NLP models and (2) NLP-Opt: three end-to-end optimization techniques to accelerate the identified performance-critical operations on various hardware platforms (e.g., CPU, GPU, FPGA). In this way, NLP-Fast can accelerate various types of NLP models on different hardware platforms by identifying their critical operations through NLP-Perf and applying the NLP-Opt's holistic optimizations. We evaluate NLP-Fast on CPU, GPU, and FPGA, and the overall throughputs are increased by up to 2.92×, 1.59×, and 4.47× over each platform's baseline. We release NLP-Fast to the community so that users are easily able to conduct the NLP-Fast's analysis and apply NLP-Fast's optimizations for their own NLP applications.