Neural network accelerators (e.g., TPUs) have become mainstream devices in computing systems. Unfortunately, the existing accelerator-based systems for neural networks fail to fully leverage the acceleration opportunities due to thelimited flexibility. Specifically, the majority of the accelerators focus on only the compute-intensive operations of neural networks (e.g., convolution and fully-connected layers). However, we identify that sub-optimal handling ofauxiliary operationssuch as embedding and compression can incur non-trivial loss in terms of accuracy, training speed, and adaptability to new domains. The problem persists considering that recent advancements in neural networks often come from auxiliary operations. To effectively handle rapidly evolving auxiliary operations and maximize acceleration opportunities, we propose DLS, a holistic neural network acceleration system using heterogeneous computing devices. The key idea is to distribute compute-intensive operations on highly specialized ASICs for maximum performance, and auxiliary operations on flexible devices (e.g., FPGA, GPU) for better adaptability. We emphasize that a naïve integration of different devices fails to deliver high performance due to high communication overheads. To address this communication inefficiency, we propose an efficient FPGA-based device orchestration utilizing direct device-to-device communication and fine-grained operation scheduling. In this way, our system alleviates the communication overhead between heterogeneous devices by removing expensive kernel stack traversal and leveraging computation units and communication links in parallel. The evaluation using popular neural networks with emerging auxiliary operations shows that our system achieves both flexibility and high performance for various cases from single-accelerator training to distributed training (2.6–8.9× speedup).

Neural network is a major driving force of another golden age of computing; the computer architects have proposed specialized accelerators (e.g., TPU), high-speed interconnects (e.g., NVLink), and algorithms (e.g., ring-based reduction) to efficiently support neural network applications. As a result, they achieve orders of magnitude higher efficiency for neural network computation and inter-accelerator communication over traditional computing platforms.In this paper, we identify that the emerging platforms have shifted the performance bottleneck of neural network from model computation and inter-accelerator communication to data preparation. Although overlapping data preparation and the others has hidden the preparation overhead, the higher input processing demands of emerging platforms start to reverse the situation; at scale, data preparation requires an infeasible amount of the host-side CPU, memory, and PCIe resources. Our detailed analysis reveals that this heavy resource consumption comes from data transformation for neural network specific formats, and buffering for communication among devices.Therefore, we propose a scalable neural network server architecture by balancing data preparation and the others. To achieve extreme scalability, our design relies on a scalable device array, rather than the limited host resources, with three key ideas. First, we offload CPU-intensive operations to the customized data preparation accelerators to scale the training performance regardless of the host-side CPU performance. Second, we apply direct inter-device communication to eliminate unnecessary data copies and reduce the pressure on the host memory. Lastly, we cluster underlying devices considering unique communication patterns of the neural network processing and interconnect characteristics to efficiently utilize aggregated interconnect band-width. Our evaluation shows that the proposed architecture achieves 44.4× higher training throughput on average over a naively extended server architecture with 256 neural network accelerators.