VTA-NIC Chip ArchitectureWe aim to achieve DL inference serving (DLIS) without CPU interference.We integrate hardware data paths as a NIC (Network Interface Card), a REST API parser/deparser and multiple VTAs (Versatile Tensor Accelerators).ConfigurationVTA-NICProcess node16 nm FinFET @Xilinx FPGANumber of Cores8 VTA CoresCore Frequency213 MHzMACs per core169Memory Throughput19.2GB/s (DDR4-2400)Number precisionINT8PerformancePower EfficiencyThe DLIS power efficiency of VTA-NIC is 6.1x better than that of GPU (Nvidia V100).Tail LatencyAt high load, the tail latency of heterogeneous systems unexpectedly increases. With our chip, the tail latency is predictable since it is proportional to the load.BackgroundRecently, web applications are often built on microservices.DL Inference Serving (DLIS) is one of those microservices1.DLIS is provisioned with a special accelerator instance2.The microservices/instances are loosely coupled via APIs.BackgroundAccelerator instances risk inefficient data movement.1.Moving data via host processors decreases the accelerator's utilization.3a.In our preliminary experiments, half of the DLIS latency was caused by moving data.2.Under high-load conditions, the interference of host processors degrades DLIS tail latency by up to 100 times4.a.In the real cloud, 9% of light DLIS tasks suffer server tail latency, and half of the serving time is waiting time.5Model: ResNet-18@TensorRTPrecision: INT8System: Triton Inference ServerAccelerator: Nvidia V100

Divide-and-conquer algorithms are suitable for modern parallel machines, tending to have large amounts of inherent parallelism and working well with caches and deep memory hierarchies. Among others, list homomorphisms are a class of recursive functions on lists, which match very well with the divide-and-conquer paradigm. However, direct programming with list homomorphisms is a challenge for many programmers. In this paper, we propose and implement a novel systemthat can automatically derive cost-optimal list homomorphisms from a pair of sequential programs, based on the third homomorphism theorem. Our idea is to reduce extraction of list homomorphisms to derivation of weak right inverses. We show that a weak right inverse always exists and can be automatically generated from a wide class of sequential programs. We demonstrate our system with several nontrivial examples, including the maximum prefix sum problem, the prefix sum computation, the maximum segment sum problem, and the line-of-sight problem. The experimental results show practical efficiency of our automatic parallelization algorithm and good speedups of the generated parallel programs.