The demand for virtual private networks (VPNs) that provide confidentiality, integrity, and authenticity of communications is growing every year. IPsec is one of the oldest and most widely used VPN protocols, implemented between the internet protocol (IP) layer and the data link layer of the Linux kernel. This implementation method, known as bump-in-the-stack, has the advantage of being able to transparently apply IPsec to traffic without changing the application. However, its throughput efficiency (Gbps/core) is worse than regular Linux communication. Therefore, we chose the bump-in-the-wire (BITW) architecture, which handles IPsec in hardware separate from the host. Our proposed BITW architecture consists of inline cryptographic accelerators implemented in field-programmable gate arrays and a programmable switch that connects multiple such accelerators. A VPN gateway implemented with our architecture is transparent and improves the throughput efficiency by 3.51 times and power efficiency by 3.40 times over a VPN gateway implemented in the Linux kernel. It also demonstrates excellent scalability, and has been confirmed to scale to a maximum of 386.24 Gbps per tunnel, exceeding state-of-the-art technology in maximum throughput and efficiency per tunnel. In multiple-tunnels use cases, the proposed architecture improves the energy efficiency by 2.49 times.

VTA-NIC Chip ArchitectureWe aim to achieve DL inference serving (DLIS) without CPU interference.We integrate hardware data paths as a NIC (Network Interface Card), a REST API parser/deparser and multiple VTAs (Versatile Tensor Accelerators).ConfigurationVTA-NICProcess node16 nm FinFET @Xilinx FPGANumber of Cores8 VTA CoresCore Frequency213 MHzMACs per core169Memory Throughput19.2GB/s (DDR4-2400)Number precisionINT8PerformancePower EfficiencyThe DLIS power efficiency of VTA-NIC is 6.1x better than that of GPU (Nvidia V100).Tail LatencyAt high load, the tail latency of heterogeneous systems unexpectedly increases. With our chip, the tail latency is predictable since it is proportional to the load.BackgroundRecently, web applications are often built on microservices.DL Inference Serving (DLIS) is one of those microservices1.DLIS is provisioned with a special accelerator instance2.The microservices/instances are loosely coupled via APIs.BackgroundAccelerator instances risk inefficient data movement.1.Moving data via host processors decreases the accelerator's utilization.3a.In our preliminary experiments, half of the DLIS latency was caused by moving data.2.Under high-load conditions, the interference of host processors degrades DLIS tail latency by up to 100 times4.a.In the real cloud, 9% of light DLIS tasks suffer server tail latency, and half of the serving time is waiting time.5Model: ResNet-18@TensorRTPrecision: INT8System: Triton Inference ServerAccelerator: Nvidia V100